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The  NCLR  Center  for  Health  Promotion  provides  technical  assistance  to  NCLR  affiliates 
and  other  national,  regional,  and  local  entities  which  share  a  commitment  to  health-related 
education  and  disease  prevention  in  the  Hispanic  community.  The  NCLR  Center  for  Health 
Promotion  houses  the  following  initiatives: 

♦  The  NCLR  AIDS  Center,  established  in  1988  through  a  grant  from  the  Centers 
for  Disease  Control  and  Prevention,  carries  out  interrelated  activities  built  around  the  concept 
of  an  interactive  HIV/AIDS  network:  information  sharing,  development  of  guides  and  manuals, 
development  of  program  models,  training,  and  capacity-building  technical  assistance. 

♦  The  Hispanic  Health  Liaison  Project,  established  in  the  fall  of  1991  through  a 
grant  from  the  Office  of  Minority  Health,  is  designed  to  provide  information,  assistance,  and 
support  to  Hispanic  community-based  organizations  committed  to  increasing  Hispanic 
involvement  in  preventive  health  efforts,  and  serve  as  a  bridge  between  community-based 
organization  and  mainstream  agencies,  public  and  private.  A  major  product  of  the  project  will 
be  a  comprehensive  report  on  Hispanic  health  status  designed  to  increase  awareness  and 
understanding  of  Hispanic  health  needs  among  policy  makers,  mainstream  organizations,  the 
media,  the  Hispanic  community,  and  the  general  public. 

♦  Special  health  initiatives  such  as  an  analysis  of  Hispanic  health  insurance  coverage 
and  seminars  and  conference  workshops  on  substance  abuse,  HIV/ AIDS,  and  other  health  issues. 
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The  National  Council  of  La  Raza  (NCLR),  the  largest  constituency-based  national  Hispanic 
organization,  exists  to  improve  life  opportunities  for  the  more  than  22  million  Americans  of 
Hispanic  descent.  In  addition  to  its  Washington,  D.C.  headquarters,  NCLR  maintains  field  offices 
in  Los  Angeles,  California;  Phoenix,  Arizona;  McAllen,  Texas;  and  Chicago,  Illinois.  NCLR  has 
four  missions:  applied  research,  policy  analysis,  and  advocacy  on  behalf  of  the  entire  Hispanic 
community;  capacity-building  assistance  to  support  and  strengthen  Hispanic  community-based 
organizations;  public  information  activities  designed  to  provide  accurate  information  and 
positive  images  of  Hispanics;  and  special  innovative,  catalytic,  and  international  projects.  NCLR 
acts  as  an  umbrella  for  160  affiliates  —  Hispanic  community-based  organizations  which  together 
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I.  INTRODUCTION 


A.  About  this  Manual 

This  guide,  Understanding  Evaluation  Techniques:  The  Building  Blocks  of  Evaluation, 
is  part  of  an  ongoing  effort  by  the  NCLR  AIDS  Center  to  assist  Hispanic  community-based 
organizations  evaluate  their  HIV/STD  education  and  prevention  programs.  It  supplements 
the  NCLR  AIDS  Center’s  earlier  manual,  Evaluating  AIDS  /STD  Education  and  Prevention 
Programs:  An  Introduction.  The  first  manual  was  developed  to  help  project  staff  take  the 
first  steps  in  planning  and  carrying  out  necessary  and  practical  types  of  documentation  and 
evaluation  of  HIV/STD  prevention  programs.  This  manual  expands  on  that  effort  by 
focusing  on  basic,  cost-effective  program  evaluation  techniques  which  can  provide  valuable 
information  about  the  effectiveness  of  each  step  of  an  HIV/STD  prevention  program.  This 
manual  also  provides  general  guidance  on  determining  what  kinds  of  evaluation  results  are 
meaningful  and  obtainable  in  a  community  setting  and  strategies  that  are  appropriate  and 
sensitive  to  the  cross-cultural  populations  served  by  community-based  organizations, 
especially  Hispanic  and  other  minority-oriented  agencies. 

The  intended  users  of  this  guide  are  professionals  involved  in  HIV/STD  issues  or  in 
delivering  HIV/STD  education  and  prevention  services  to  Hispanic  and  other  minority 
groups,  managerial  and  field  staff  of  AIDS  programs  in  Hispanic  community-based 
organizations,  their  public-  and  private-sector  funders,  community  and  migrant  health 
centers,  and  other  AIDS  and  human  services  providers. 

The  evaluation  techniques  described  in  this  manual  include:  use  of  experts,  focus 
groups,  community  interviews,  demonstration  sessions,  observers,  informal  surveys,  and 
games.  These  techniques  can  be  used  individually  to  achieve  a  successful  evaluation. 
However,  as  community-based  organizations  become  comfortable  using  one  or  two 
techniques,  they  are  encouraged  to  build  in  additional  ones  for  a  more  comprehensive 
evaluation  plan.  For  community-based  organizations  with  little  or  no  evaluation  experience, 
this  manual  provides  a  set  of  concepts  and  strategies  necessary  for  beginning  program 
evaluation.  For  organizations  with  more  evaluation  experience,  this  guide  should  provide 
insight  into  the  strengths  and  weaknesses  of  current  evaluation  practices  and  ideas  for  broader 
approaches. 

The  techniques  described  are  ones  which  NCLR  has  found  effective  and  do  not 
include  all  possible  methods  for  evaluation.  Although  the  information  and  terminology  is  in 
accordance  with  the  Centers  for  Disease  Control  and  Prevention  (CDC)  guidelines,  this 
manual  reflects  NCLR's  preferred  methods  and  should  not  be  viewed  as  the  only  model  for 
evaluation. 
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Community-based  organizations  are  urged  to  begin  to  develop  comprehensive 
evaluation  plans  for  their  HIV/STD  programs.  Once  organizations  gain  practice  in  using  the 
evaluation  techniques  described  in  this  manual,  they  will  discover  that  program  evaluation  is 
feasible,  manageable,  and  beneficial. 

B.  The  Need  for  Evaluation 

The  HIV/AIDS  epidemic,  now  in  its  second  decade,  continues  to  disproportionately 
affect  the  Hispanic  community.  In  the  absence  of  a  proven  cure  for  AIDS,  the  need  to 
continue  carrying  out  HIV/STD  education  and  prevention  activities,  outreach  to  groups 
engaging  in  high-risk  behaviors,  and  other  AIDS-related  services  is  clear.  What  remains 
unclear  are  what  types  of  programs  and  interventions*  are  effective  in  reducing  the  spread  of 
HIV/STDs.  The  implementation  of  new  health  education  and  risk  reduction  (HERR) 
activities  and  expansion  of  existing  ones  will  increasingly  be  contingent  on  the  ability  of 
organizations  to  evaluate  the  benefits  of  their  efforts.  If  organizations  do  not  evaluate  their 
programs,  there  is  no  way  of  knowing  whether  interventions  are  effectively  serving  the  needs 
of  clients,  or  where  improvements  should  be  made.  To  date,  little  data  on  the  effectiveness 
of  primary  prevention  programs  has  been  generated. 

Ongoing  program  evaluation  can  assist  in  identifying  effective  strategies  or 
interventions  to  reduce  the  incidence  of  HIV/STD  disease.  Evaluation,  through  such 
approaches  as  focus  groups  and  informal  surveys,  reveals  what  benefits  to  the  client,  if  any, 
result  from  an  intervention.  It  provides  information  for  the  program  operator  to  modify 
approaches  which  appear  ineffective  and  incorporate  interventions  which  are  better  targeted 
to  the  intended  groups.  When  regularly  done,  evaluation  can  provide  valuable  data  on 
changing  community  and  program  needs.  Informal  surveys  collected  from  a  group  of  high 
school  students  can  indicate  a  greater  need  for  "AIDS  102"  than  for  basic  "AIDS  101" 
education  sessions.  It  can  identify  important  issues  or  populations  not  being  addressed  by  a 
program,  staff  and/or  client  concerns  that  program  directors  may  not  have  been  aware  of, 
and  other  obstacles.  The  actions  taken  as  a  result  of  carrying  out  program  evaluation  can 
enhance  the  quality  of  an  HIV/STD  program  for  program  staff  and  the  clients  of  the 
organization. 

Hispanic  community-based  organizations,  Hispanic-controlled  institutions  committed 
to  the  socioeconomic  and  health  advancement  of  Hispanics,  play  a  critical  role  in  raising  the 
awareness  of  policy  makers  about  the  need  for  HIV/STD  prevention  activities.  The  Hispanic 
community,  expected  to  become  the  largest  minority  population  in  the  United  States  by  the 
turn  of  the  century,  faces  many  problems  which  are  not  well  documented,  and  therefore  not 


Intervention,  a  term  used  frequently  throughout  this  manual,  refers  to  any  planned  activity 
designed  to  produce  intended  changes  in  a  target  population.  Some  examples  are  HIV/STD 
education  workshops,  home-based  HIV/STD  "101"  sessions,  street  level  outreach,  etc. 
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well  understood.  AIDS  is  a  major  problem  in  the  Hispanic  community,  and  one  which  is 
interrelated  with  many  other  issues  facing  Hispanics,  such  as  high  poverty  and  low 
educational  attainment.  One  way  community-based  organizations  can  educate  policy  makers 
and  other  organizations  about  the  importance  and  benefits  of  providing  community-focused, 
culturally  appropriate  HIV/STD  programs  is  through  program  evaluation.  Because 
community-based  organizations  have  direct  contact  with  individuals  that  practice  behaviors 
placing  them  at  risk  of  HIV/STDs  and  who  would  not  be  reached  otherwise  with  AIDS 
education  efforts,  these  organizations  possess  invaluable  information  about  risk  behaviors, 
trends,  and  the  types  of  HIV/STD  interventions  most  needed  in  the  community.  Community- 
based  organizations  are  in  an  ideal  position  to  educate  policy  makers  and  government 
officials  about  the  importance  and  need  for  community-based  interventions.  However,  policy 
makers  want  information  which  reflects  systematic  documentation  and  assessment  of 
community-based  efforts. 

Funders  of  community-based  organizations  are  interested  in  evaluation  because  this 
provides  assurance  that  their  funds  are  being  spent  on  effective  activities.  It  is  also  assuring 
to  know  that  funds  are  going  to  organizations  which  share  common  goals  and  care  about  the 
quality  their  work.  A  carefully  thought  out  and  clearly  written  evaluation  plan  increases  the 
likelihood  that  a  project  proposal  will  satisfy  a  major  funding  criteria. 

C.  The  Growing  Problem  of  AIDS  and 

Other  STDs  in  the  Hispanic  Community 

The  prevalence  of  STDs  among  minority  groups  continues  to  be  a  major  concern. 
Some  of  the  most  threatening  STDs  include  HIV/AIDS,  gonorrhea,  syphilis,  chlamydia,  and 
pelvic  inflammatory  disease.  Because  the  same  sexual  behaviors  that  place  individuals  at  risk 
of  HIV/AIDS  can  also  place  one  at  risk  of  other  STDs,  AIDS  education  and  outreach 
activities  should  include  education  on  non-HIV  STDs  as  well. 

HIV/AIDS  is  by  far  the  deadliest  sexually  transmitted  disease.  AIDS  cases  reported 
in  the  latter  part  of  the  decade  have  shown  that  this  disease  can  strike  any  human  being, 
regardless  of  age,  gender,  sexual  orientation,  or  geographic  location.  A  disease  that  a 
decade  ago  was  viewed  as  affecting  primarily  homosexual  White  men  is  now  a  threat  to  all 
population  segments,  and  is  affecting  minority  groups  the  hardest. 

The  disproportionate  effect  of  AIDS  on  Hispanics  is  evident.  Comprising  about  9% 
of  the  U.S.  population,  Hispanics  represent  16%  of  total  AIDS  cases  as  of  June  1992. 
Hispanic  women  are  less  than  10%  of  the  U.S.  female  population,  but  make  up  20%  of  total 
female  AIDS  cases.  HIV  disease  in  pregnant  mothers  often  leads  to  infection  among 
Hispanic  children,  who  represent  25%  of  all  pediatric  AIDS  cases.  Nor  have  adolescents 
been  spared;  Hispanic  adolescents,  accounting  for  12%  of  13-19  year-olds  in  the  U.S. 
comprise  19%  of  reported  AIDS  cases  among  teenagers.1  Many  more  teenagers  are  possibly 
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HIV-infected  but  unaware  of  it.  Because  the  virus  can  remain  asymptomatic  for  many  years, 
the  magnitude  of  HIV  transmission  among  today’s  teenagers  will  not  be  fully  realized  for 
years  to  come. 

The  threat  posed  by  other  sexually  transmitted  diseases  must  not  be  overlooked  in 
HERR  activities.  Not  only  are  transmission  modes  similar;  STD-infected  persons  are  also 
highly  susceptible  to  HIV/ AIDS.  The  lesions  and  open  sores  often  caused  by  STDs  allow 
easy  entry  of  HIV  into  the  bloodstream.  The  media  attention  given  to  HIV/AIDS  has 
overshadowed  the  threat  of,  and  the  interrelationship  between  AIDS  and  other  STDs.  Yet 
STD  rates  among  minorities  remain  high  compared  to  rates  among  Whites.  For  instance,  the 
1990  gonorrhea  rate  for  Hispanics  was  127.4  per  100,000  compared  to  45.8  per  100,000 
among  Whites.  Among  Blacks,  the  rate  was  39  times  higher,  1,792.1  per  100,000.  Primary 
and  secondary  syphilis  (PSS)  has  reached  epidemic  levels.  The  total  PSS  rate  for  1990  was 
20.4  per  100,000,  the  highest  rate  in  40  years.  Syphilis  rates  among  Hispanics  are  seven 
times  as  high  as  among  Whites  (16.8  per  100,000  versus  2.4  per  100,000  in  1990).  Among 
adolescents  and  young  adults,  gonorrhea  rates  are  extremely  high.  The  1990  gonorrhea  rate 
for  15-19  year-olds  was  1,060.6  per  100,000;  among  20-24  year-olds,  the  rate  was  1,089.9, 
compared  to  525.2  for  25-29  year-olds.2  Teenagers  and  young  adults,  particularly  minority 
women,  suffer  from  other  STDs  as  well,  including  chlamydia,  herpes  simplex  virus,  and 
human  papillomavirus. 

Clearly,  STD  education  and  prevention  messages  must  be  integrated  with  HIV 
prevention  messages.  Although  HIV  is  currently  fatal,  the  risk  of  acquiring  other  STDs  is 
much  greater  than  acquiring  HIV.  Also,  many  of  the  same  strategies  used  to  prevent  the 
transmission  of  other  STDs  are  effective  in  decreasing  the  risk  of  HIV  infection. 


Additional  STD  information  can  be  obtained  by  contacting  the  National  STD  Hotline  at 
1-800-227-8922.  Other  informational  sources  are  the  CDC’s  Division  of  STD  /HIV 
Prevention  1991  Annual  Report  and  the  STD  Surveillance  1991  report.  These  can  be 
obtained  by  contacting  CDC’s  Surveillance  and  Information  Systems  Branch  at 
(404)  639-0511.  You  may  also  contact  the  NCLR  AIDS  Center  to  receive  a  copy  of 
Adolescents  and  HIV/STD ,  one  of  several  fact  sheets  developed  and  updated  annually  by 
the  NCLR  Center  for  Health  Promotion. 


D.  The  Importance  of  Program  Evaluation 

Many  Hispanic  community-based  organizations  are  providing  HIV/STD-related 
services  essential  to  slowing  the  spread  of  these  diseases.  HIV/STD  prevention  and 
education  sessions,  community  outreach  campaigns,  case  management,  HIV  and  other  STD 
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counseling  and  testing,  support  group  services,  and  street  outreach  are  some  of  the  major 
activities  community-based  organizations  are  undertaking  to  reduce  the  incidence  of  new 
HIV/STD  cases.  As  these  organizations  continue  to  broaden  the  scope  of  HIV/STD 
programs  and  gain  recognition  as  critical  players  in  the  AIDS  solution,  program  evaluation 
will  increase  in  importance.  Three  reasons  in  particular  demand  that  community-based 
organizations  pay  greater  attention  to  program  evaluation: 

•  Evaluation  is  necessary  to 
assess  the  progress  and 
effectiveness  of  HIV/STD- 
related  activities.  It  helps  to 
ensure  that  promising  approaches 
be  continued,  strengthened, 
documented,  and  disseminated, 
and  ineffective  approaches  be 
changed.  Evaluation  is 
especially  important  for  new 
projects  and  models.  A  good 
assessment  indicates  the  extent  to 
which  activities  originally  stated 
in  the  project  work  plan  were 
carried  out.  If  some  activities 
changed  or  if  target  populations 
are  different,  ongoing  evaluation 
can  reveal  and  substantiate  the 
reasons  for  these  changes  or  the 
need  for  a  different  approach. 

•  Systematic  evaluation  benefits 
the  overall  operation  of  a 
program.  Evaluation 
encourages  staff  to  think  realistically  about  the  activities  required  in  a  project,  and 
whether  these  are  in  fact  achievable  in  the  timetable  specified.  It  mandates  short-  and 
long-term  planning,  thereby  promoting  proactive  versus  reactive  approaches  to 
program  development. 

•  A  clearly-defined  evaluation  plan  can  make  the  difference  between  a  project 
proposal’s  approval  or  rejection.  As  more  community-based  organizations  seek 
first-time  funding  for  AIDS-  and  other  STD-related  projects  or  refunding  for  existing 
projects,  writing  clear  evaluation  plans  for  proposed  objectives  and  demonstrating  the 
capacity  to  carry  out  such  plans  will  be  critical.  Only  through  analyzing  the  program 
evaluation  plan  can  funders  understand  how  funds  are  being  spent  and  the  benefits  of 
funding  one  community  group’s  education  and  prevention  program  versus  another. 
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COMMUNITY-BASED 
ORGANIZATIONS  NEED  TO 
EVALUATE  HIV/STD  PROGRAMS 

•  To  monitor  the  progress  and 
effectiveness  of  project  activities, 
make  necessary  changes  or  revisions, 
and  contribute  to  the  "state  of  the 
art"  of  HIV/STD  prevention  efforts. 

•  To  benefit  the  overall  operation  of  a 
program  by  encouraging  proactive, 
long-  and  short-term  planning  of 
activities  and  minimizing  reactive 
approaches. 

•  To  provide  project  funders  evidence 
of  how  funds  are  being  spent  and 
thus  demonstrate  the  benefits  of 
community-based  HIV/STD 
programs. 


Many  HIV/STD  programs  of  community-based  organizations  are  still  relatively  new, 
therefore  designing  and  implementing,  or  enhancing  an  evaluation  plan  should  not  be  as 
cumbersome  as  developing  one  for  a  long-established  program.  However,  key  to  a 
successful  evaluation  is  program-wide  commitment  to  the  evaluation  process.  This  entails 
understanding  program  goals  and  objectives;  understanding  the  needs,  trends,  and  risks  of  the 
target  population;  developing  appropriate  forms  and/or  surveys  to  obtain  information; 
training  staff  on  basic  community-oriented  evaluation  techniques  such  as  focus  groups  and 
community  interviews;  and  documenting  all  findings.  Corporations  place  a  high  priority  on 
these  approaches  when  developing  and/or  assessing  new  products  or  marketing  plans,  and 
community-based  organizations  must  begin  to  do  the  same. 

E.  Barriers  to  Effective  Evaluation  Encountered 
by  Community -Based  Organizations 

Community-based  organizations  face  barriers  in  conducting  program  evaluation  for 
several  reasons.  These  organizations  exist  to  provide  assistance  to  individuals  in  need,  many 
of  whom  would  not  go  to  traditional  settings  for  assistance  (e.g.,  runaway  youth,  drug 
addicts,  or  undocumented  immigrants).  Evaluating  knowledge  and  behavior  changes  among 
these  groups  is  often  challenging  due  to  the  difficulty  in  maintaining  regular  contact  with 
them.  Because  community-based  organizations  are  often  the  first  and  primary  source  of  any 
type  of  health-related  assistance  for  many  Hispanics,  heavy  emphasis  is  placed  on  such 
services  as  client  education,  counseling,  outreach,  information  dissemination,  and  client 
referrals.  These  organizations,  which  often  have  limited  staff  to  do  follow-up  on  clients, 
seldom  see  the  product  of  their  efforts.  This  makes  it  very  difficult  for  community-based 
organizations  to  measure  behavior  change  directly,  and  may  even  discourage  organizations 
from  conducting  any  evaluation  at  all.  Also,  community-based  organizations  often  feel  under 
considerable  pressure  to  provide  the  community  they  serve  with  the  most  comprehensive 
level  of  services  in  spite  of  limited  funds.  Due  to  the  reactive  nature  of  community-based 
organizations’  work  and  emphasis  on  service  provision,  evaluation  planning  has  received 
limited  attention.  In  addition,  community-based  organizations  many  times  lack  the  financial 
resources  and  staff  needed  to  carry  out  effective  evaluation. 

In  spite  of  all  that  has  been  written  on  the  subject  of  evaluation,  little  of  the  available 
material  is  useful  to  community-based  organizations.  Much  of  the  evaluation  literature 
focuses  on  costly  research  methods  designed  to  show  statistical  changes  in  health  outcomes 
resulting  from  intervention  programs.  Moreover,  the  available  data  on  evaluation  are  often 
complex.  The  use  of  evaluation  terminology  is  inconsistent,  and  varies  from  one  report  to 
another.  The  wide  diversity  of  HIV/STD-related  projects  obviously  complicates  the  task  of 
designing  primary  prevention  evaluation  manuals.  Nevertheless,  increased  efforts  must  be 
directed  at  designing  technical  assistance  manuals  appropriate  for  use  with  and  by  staff  of 
community-based  organizations.  The  National  Council  of  La  Raza  (NCLR)  AIDS  Center  is 
committed  to  assisting  with  this  process. 
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F.  Organization  of  Manual 


This  guide  contains  several  sections: 

Section  I  provides  a  general  introduction  to  the  manual. 

Section  II  examines  the  importance  of  project  goals  and  objectives  in  program 
evaluation;  objectives  are  defined  in  some  detail. 

Section  III  discusses  the  importance  of  monitoring  and  assessing  all  phases  of  a 
prevention  project  or  activity.  This  involves  evaluating  the  design,  implementation,  and  end 
of-project  results. 

Section  IV  describes  different  methods  for  evaluating  HIV/STD  education  and 
prevention  efforts.  These  include  needs  assessments,  expert  reviews,  community  input,  post 
session  assessments,  pre-  and  post-tests,  informal  surveys,  games,  case  studies,  role  plays, 
key-person  interviews,  and  other  methods. 

Section  V  illustrates  how  various  techniques  can  be  applied  at  any  stage  of  a  project 
in  order  to  assess  the  many  areas  that  comprise  an  effective,  well-run  program.  Three 
HIV/STD  education  and  prevention  programs  are  used  as  examples:  media  and  community 
outreach  campaigns,  informal  or  street  outreach  interventions,  and  HIV  counseling  and 
testing  programs. 

Section  VI  is  the  conclusion  of  the  manual. 
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II.  THE  IMPORTANCE  OF  PROGRAM  GOALS  AND 
OBJECTIVES  IN  DEVELOPING 
AN  EVALUATION  PLAN 

Program  evaluation  cannot  occur  without  clearly  defined  goals  and  objectives.  Before 
a  project  intervention  is  designed  and  implemented,  the  program  goals  and  objectives  must  be 
cleajly  identified  and  stated  in  writing.  Not  only  are  they  a  significant  part  of  the  project 
proposal  to  funders,  but  these  goals  and  objectives  also  serve  as  a  road  map  for  project 
activities  and  as  a  standard  for  measuring  success.  The  objectives,  especially,  provide  the 
basis  for  evaluation;  they  specify  what  the  program  is  supposed  to  accomplish. 

A.  Goals 

A  goal  is  a  general  statement  of  what  the  project  hopes  to  accomplish,  or  to 
contribute  toward  accomplishing,  in  the  long  term.  It  seldom  has  a  time  deadline  attached, 
and  may  not  be  achievable  through  the  efforts  of  a  single  project,  although  several  projects 
or  efforts  together  should  contribute  towards  the  goal.  Often  progress  towards  a  goal  can  be 
determined  only  after  some  years  of  effort.  A  goal  statement  should  be  both  challenging  and 
attainable.  All  HIV  education  and  prevention  programs  include  some  aspect  of  reduction  in 
HIV  transmission  as  their  long-term  goal.  Examples  of  goals: 

•  To  reduce  the  incidence  of  HIV/AIDS  among  young  Hispanic  men  in  East 
Austin,  Texas. 

•  To  reduce  HIV  transmission  through  needle-sharing  among  injecting  drug 
users  in  Chicago. 

•  To  ensure  that  Hispanics  in  Boston  who  engage  in  high-risk  behaviors  for  HIV 
infection  obtain  HIV  counseling  and  testing. 

In  general,  goal  statements  are  much  less  complex  to  write  than  objectives  because 
goals  need  not  provide  expected  project  results  in  measurable  terms.  This  specificity  is 
required  of  objectives.  (Since  funders  often  have  very  specific  funding  criteria,  the 
requirements  for  goals  and  objectives  may  vary.  Always  check  with  the  funder  when  the 
meanings  of  goals  and  objectives  are  unclear.) 

B.  Objectives 

Objectives  are  the  near-term  or  intermediate  accomplishments  which  contribute  to  the 
attainment  of  the  goal.  Objectives  have  the  following  attributes;  they  are:3 
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•  Specific:  Identify  who,  what,  and  where.  Who  is  conducting  the 
intervention?  Who  is  the  target  population  of  the  proposed  intervention?  Be 
sure  to  clearly  define  this  population  (e.g.,  persons  with  HIV/AIDS, 
incarcerated  women,  etc.).  What  type  of  intervention  or  product  will  be 
accomplished?  Where  will  the  activity  or  intervention  take  place? 

•  Measurable:  Identify  when  and  how  many;  specify  both  a  timeframe  for  the 
completion  of  a  project  or  activity  and  the  number  of  clients  to  be  served  or 
services  to  be  provided  (make  sure  your  stated  activities  or  results  can  be 
measured  to  demonstrate  that  your  planned  activities  were  carried  out  or 
desired  outcomes  achieved).  Measurement  is  the  essence  of  evaluation  and 
program  planning  is  the  place  where  evaluation  is  built  into  the  project. 

•  Reasonable:  Can  be  achieved  within  the  specified  time  using  available 
technology  and  resources.4 

Objectives  for  carrying  out  a  long-term  HIV/STD  project  are  better  understood  and 
appreciated  when  classified  by  type.  The  time  invested  in  breaking  down  objectives  by  type 
in  the  beginning  of  a  project  will  not  only  enhance  the  appearance  of  the  project  proposal, 
but  will  enable  better  management  of  activities,  and  hopefully  minimize  the  occurrence  of 
problems  related  to  lack  of  planning  during  the  project. 

There  are  two  major  types  of  objectives: 

•  Outcome  objectives:  An  outcome  objective  states  the  desired  results  of  a 
program  or  intervention  in  measurable  terms.  It  should  be  specific,  realistic, 
and  achievable  during  the  project  period.  (See  box  on  the  next  page  for 
suggestions  on  writing  outcome  objectives.)  For  an  HIV-related  prevention 
and  education  project,  results  are  usually  stated  in  quantitative  terms  (e.g.,  a 
targeted  level  of  client  returns  for  test  results  within  an  HIV  testing  site,  a 
specified  increase  in  condom  sales  in  a  particular  neighborhood).  Examples  of 
outcome  objectives: 

♦  To  increase  from  40%  to  75%  within  one  year  the  proportion  of 
individuals  at  La  Clinica’s  anonymous  HIV  testing  site  who  return  for 
test  results  and  counseling. 

♦  To  increase  from  40%  to  80%  the  proportion  of  junior  high  school 
students  from  the  South  Bronx  who  answer  correctly  seven  out  of  ten 
questions  on  an  HIV/STD  KABB  survey  by  September  1,  1994. 


NCLR  AIDS  Center 
Understanding  Evaluation  Techniques 


9 


♦  To  increase  from  20%  to  50%  within  three  years  the  percentage  of 
injecting  drug  users  (IDUs)  attending  South  Side  Drug  Treatment 
Center  who  report  using  bleach  and  water  almost  always  to  disinfect 
their  "works." 


SUGGESTIONS  FOR  WRITING  OUTCOME  OBJECTIVES 

•  Begin  with  the  word  "to,"  followed  by  an  action  verb. 

•  In  each  objective,  specify  just  one  major  result  to  be  accomplished. 

•  Define  the  target  population  of  the  proposed  intervention. 

•  Specify  a  target  date  for  completion. 

•  State  "what"  will  be  achieved  and  by  "when"  —  it  is  not  necessary  to  state 
"why"  and  "how." 

•  Be  sure  the  objective  is  measurable  so  that  you  can  verify  whether  or  not  it 
has  been  met. 

•  Be  sure  the  objective  is  attainable  given  the  agency’s  resources. 


•  Process  objectives:  A  process  objective  specifies  the  completion  of  key 
activities  or  tasks  essential  to  achieving  an  outcome.  Taken  as  a  group, 
process  objectives  are  steps  towards  the  desired  results.  A  process  objective 
must  be  specific,  realistic,  measurable,  and  time-phased.  Examples  of  process 
objectives: 

♦  To  assure  that  50  high  school  parents  complete  a  six-hour  training 
course  on  HIV/STD  disease  and  transmission  within  six  months  after 
project  initiation. 

♦  To  form  a  20-member  advisory  group  of  teachers,  social  workers,  HIV 
counselors  and  educators  to  assist  in  developing  an  HIV/STD 
curriculum  for  ninth  graders  within  three  months  after  project  initiation. 
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♦  To  conduct  48  hours  of  HIV/STD  education  to  all  ninth  grade  students 
of  Madison  High  School  within  one  full  semester  (January  through 
June). 

In  writing  an  outcome  or  process  objective,  make  sure  you  include  expected 
measurable  results.  Ask  yourself  if  you  can  measure  or  verify  whether  the  objective  has 
been  met.  If  not,  you  probably  need  to  specify  more  clearly  the  results  you  want 
accomplished,  or  add  some  criteria  for  measuring  success.  See  examples  below: 

Outcome  objective  lacking  measurable  results: 


To  develop  and  air  a  Spanish-language  AIDS  prevention  radio  public  service 
announcement  (PSA)  which  effectively  reaches  Hispanic  teenagers,  within  12 
months  after  project  initiation. 


Outcome  objective  with  measurable  criteria  added: 


To  develop  a  Spanish-language  AIDS  prevention  public  service  announcement  and 
air  it  on  three  radio  stations  at  times  when  the  audience  is  predominantly  Hispanic 
teenagers,  reaching  a  total  audience  of  at  least  200,000  based  on  industry  ratings, 
within  12  months  after  project  initiation. 


When  developing  your  program  objectives,  it  may  be  useful  to  list  the  measures  to  be  used  in 
determining  success,  as  shown  in  the  box  on  the  next  page: 
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GOAL/OBJECTIVES 

CRITERIA  FOR  MEASURING 
ACCOMPLISHMENT  OF  OBJECTIVES 

Goal:  To  increase  knowledge  about 
HIV/AIDS  prevention  among  Hispanic 
teenagers. 

Outcome  Objective:  To  air  an  AIDS 
prevention  PSA  that  reaches  at  least 
200,000  people  consisting  primarily  of 
Hispanic  teenagers  within  12  months 
after  project  initiation. 

Process  Objectives: 

1 .  To  develop  a  Spanish-language  AIDS 
prevention  PSA  that  targets  Hispanic 
teenagers  within  three  months  after 
project  initiation. 

2.  To  contract  with  three  radio  stations 
which  agree  to  air  the  PSA  at  times 
when  their  audience  is  predominantly 
Hispanic  teenagers  within  two  months 
after  project  initiation. 

1.  A  Spanish-language  AIDS  public  service 
announcement  is  developed  and  taped. 

2.  The  PSA  is  aired  by  three  radio  stations. 

3.  The  PSA  is  aired  at  times  that  ratings 
indicate  that  the  audience  is 
predominantly  Hispanic  teenagers. 

4.  Industry  ratings  indicate  that  the  PSA  has 
reached  an  audience  of  at  least  200,000 
based  on  the  times  the  PSA  was  aired. 

5.  All  of  the  above  have  been  accomplished 

12  months  after  initiation  of  the  project. 

When  setting  outcome  objectives  especially,  be  realistic  about  what  your  HIV/STD 
intervention  program  can  accomplish  keeping  in  mind  that  these  efforts  must  in  the  short 
term  contribute  to,  or  move  towards,  changing  high-risk  behaviors.  Do  not  set  objectives 
that  cannot  be  met  or  set  objectives  that  are  really  goals.  (Remember  that  objectives  should 
be  measurable  and  obtainable.)  For  example,  a  goal  of  all  AIDS  education  efforts  is  to 
produce  changes  in  sex  and/or  drug-using  behaviors  that  reduce  the  risk  of  HIV  disease. 
Measuring  behavior  changes  is  often  difficult  and  beyond  the  scope  of  what  small 
community-based  organizations  can  do  considering  their  workload,  limited  funding  and  staff. 
Community-based  organizations  typically  do  not  follow  clients  for  a  long  period  of  time  to 
enable  them  to  assess  direct  behavior  changes.  These  organizations  can,  however, 
implement,  mobilize,  and  disseminate  AIDS  education  and  prevention  resources  which  can, 
at  some  unknown  point  in  time,  lead  to  behavior  changes.  In  the  above  example,  the 
outcome  objective  was  to  air  an  AIDS  prevention  PSA  —  an  activity  that  was  measurable, 
obtainable,  and  contributed  to  —  not  accomplished  —  the  ultimate  goal  of  reducing  high-risk 
behaviors. 
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Once  the  project  goals  and  objectives  have  been  finalized  and  have  undergone 
sufficient  staff  review,  it  may  help  to  keep  these  on  a  form  exclusively  for  goals  and 
objectives,  for  easy  reference.  If  there  are  any  questions  about  goals  and  objectives,  it  may 
be  more  efficient  to  pull  out  this  form  instead  of  pulling  out  the  entire  project  proposal.  This 
form  should  reflect  all  revisions  or  changes  made  to  the  project  objectives.  (See 
Worksheet  A.) 
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III.  TYPES  OF  PROGRAM  EVALUATION 


Program  evaluation  yields  maximum  benefit  when  it  is  applied  consistently  to  all  key 
stages  of  a  project.  In  fact,  community-based  organizations  are  strongly  encouraged  to  plan 
and  design  evaluation  strategies  prior  to  the  initiation  of  the  program,  as  part  of  the  program 
development  process.  The  following  section  examines  three  types  of  evaluation  —  design, 
process,  and  outcome  evaluation. 

A.  Design/ Formative  Evaluation 

The  term  design  evaluation,  also  referred  to  as  formative  evaluation,  describes  all 
the  evaluation  methods  used  during  and  after  program  design,  but  before  the  program  is  fully 
implemented.  Design  evaluation  also  includes  later  assessments  of  the  extent  to  which 
problems  in  implementation  were  related  not  to  the  process  but  to  flaws  in  the  original 
design.  The  information  gathered  from  this  type  of  evaluation  can  be  used  to  understand  the 
actual  need  for  a  particular  intervention,  help  guide  the  initial  design  and  implementation  of 
the  intervention,  or  improve  an  ongoing  intervention.5 

During  the  program  design  phase,  consider  some  of  the  following  questions: 

•  Why  do  we  need  this  intervention  approach  versus  another  type  of  approach? 

•  If  our  agency  is  planning  to  implement  a  teenage  pregnancy  prevention 
program,  what  key  activities  and  components  must  be  included  in  this 
program?  How  do  we  effectively  reach  this  audience?  What  staff 
qualifications  do  we  need  to  ensure  optimal  job  performance? 

•  Is  our  brochure  written  in  language  that  is  understandable  and  sensitive  to  the 
target  audience?  Are  the  messages  clear  and  accurate? 

•  What  high-risk  behaviors  are  we  trying  to  change  in  our  program?  What  do 
we  know  about  these  behaviors?  Why  do  persons  engage  in  these  behaviors? 

•  If  our  new  program  has  been  unsuccessful  in  attracting  teenagers  to  our 
prevention  sessions,  what  is  wrong?  Is  our  outreach  design  inadequate?  Are 
we  targeting  the  wrong  population?  Is  something  else  wrong  with  our  design 
—  or  are  we  not  implementing  it  properly? 

Design  or  formative  evaluation  can  be  conducted  in  many  ways.  Needs  assessments, 
expert  reviews,  focus  groups,  key-person  interviews,  and  documentation  are  possible 
approaches,  as  demonstrated  below: 
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•  Use  community  needs  assessments  to  determine  service  gaps  (e.g.,  absence 
of  culturally  and  linguistically  appropriate  AIDS  educational  materials), 
identify  target  populations  (e.g.,  recent  immigrants),  and  identify  potential 
partners  among  community-based  organizations  (e.g.,  youth  centers,  churches, 
schools,  etc.).  For  example,  information  compiled  from  a  needs  assessment 
may  reveal  the  need  for  a  community-based  organization  to  become  a  provider 
of  HIV/STD  prevention  services  for  Hispanic  teenagers. 

•  Use  expert  reviews  to  provide  feedback  on  draft  materials  and  program 
designs; 

•  Use  focus  groups  to  test  prevention  messages  or  materials.  This  could 
involve  organizing  a  group  of  teenagers  to  provide  feedback  on  a  proposed 
public  service  announcement  aimed  at  teenagers  and  using  comments  from  the 
group  to  modify  and  strengthen  the  PSA  before  it  is  aired. 

•  Use  community  interviews  and/or  key-person  interviews  to  obtain  input  on 
perceived  service  needs  and  priorities,  as  well  as  community  attitudes  about 
HIV/STDs;  and 

•  Use  documentation  of  client  demographic  information  and  program  activities 
and  its  analysis,  to  review  early  implementation  and  compare  it  with  projected 
activities  and  client  populations.  Also,  make  sure  to  thoroughly  review  and 
use  the  documented  results  of  community  needs  assessments,  focus  groups, 
expert  reviews,  and  other  evaluation  methods  to  improve  training  sessions, 
strengthen  educational  messages,  and  refine  the  overall  program. 

B.  Process  Evaluation 

Process  evaluation  describes  the  methods  that  are  used  to  assess  whether  the  project 
has  been  implemented  as  planned,  and  whether  the  short-term  process  objectives  have  been 
achieved.  It  measures  task  completion  and  efficiency.  This  type  of  evaluation  generally 
begins  with  the  implementation  of  the  intervention  and  continues  for  its  duration  to  determine 
if  interventions  and  delivery  are  functioning  as  planned  and,  if  not,  to  direct  necessary 
modifications.  Three  broad  questions  are  addressed  in  process  evaluation:  What  was  done? 
To  whom  was  it  done?  How  was  it  done?6  More  specific  questions  answered  through 
process  evaluation  might  include  the  following:7 

•  Who  is  delivering  the  intervention?  Are  they  delivering  clear,  accurate 
information?  Are  they  sensitive  to  the  needs  of  the  target  audience?  Are  they 
implementing  the  intervention  protocols  correctly? 
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•  Is  the  project  reaching  the  desired  target  groups?  Are  the  projected  number 
and  types  of  individuals  being  reached  and  served? 

•  Do  outreach  workers,  evaluators,  and  administrators  fully  understand  the 
purpose  and  scope  of  the  intervention?  Can  staff,  especially  those  providing 
direct  client  services,  agree  in  their  descriptions  of  the  interventions,  how  they 
will  be  conducted,  what  will  be  said? 

•  Are  the  tasks  in  the  work  plan  being  carried  out  on  time,  and  within  budget? 

•  Are  there  barriers  to  optimal  performance  among  intervention  workers  (e.g., 
lack  of  support  and  training,  lack  of  cultural  sensitivity,  lack  of  sufficient  staff 
and/or  lack  of  types  of  staff,  stress  and  burnout,  etc.)? 

•  How  do  clients  feel  about  the  program?  Are  the  location  and  setting 
appropriate?  Do  clients  feel  comfortable  in  discussing  the  topic?  Have  you 
documented  an  increase  in  the  number  of  clients  coming  to  your  agency  since 
you  implemented  your  new  prevention  program? 

Process  evaluation  can  be  conducted  using  a  variety  of  methods,  including  the 
following: 

•  Post-session  assessments,  pre-  and  post-tests,  informal  surveys,  games  and 
other  activities,  case  studies  and  role  plays,  and  key-person  interviews. 
Applying  one  or  more  of  these  techniques  can  assess  how  well  participants  of 
an  AIDS  workshop  understand  the  information  and/or  skills  presented  by  the 
educators; 

•  Community  indicators.  Information  such  as  the  number  of  telephone  calls 
made  weekly  to  an  AIDS  hotline,  the  number  of  new  client  visits  to  an 
HIV/STD  testing  site,  and  levels  of  condom  distribution  and  sales  following  a 
community  AIDS  awareness  campaign  provide  some  measure  of  the 
campaign’s  success  in  reaching  the  targeted  community; 

•  Administrative  recordkeeping.  This  includes  such  information  as  the  number 
of  HIV/STD  presentations/seminars  and/or  street  outreach  sessions  conducted, 
the  location  of  intervention  sessions,  names  of  key  staff  conducting 
presentations  and  street  outreach,  the  number  of  individuals  reached  through 
various  interventions,  and  demographic  characteristics  about  clients  or 
descriptions  of  the  target  groups; 
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•  Narrative  case  studies.  A  narrative  case  study  is  the  documentation  of  all  the 
major  aspects  of  the  project  from  beginning  to  end.  Background  information 
about  why  and  how  the  project  was  implemented,  the  key  staff  and/or  other 
organizations  involved  in  the  project,  the  major  planning  activities  undertaken, 
problems  encountered,  and  their  resolutions  are  some  examples  of  what  should 
be  included  in  such  a  report;  and 

•  Planned  variation.  This  method  tests  two  interventions  either  simultaneously 
or  at  different  times  —  whichever  is  most  feasible  —  for  a  defined  period  of 
time  to  find  out  which  intervention  is  most  successful  or  beneficial.  Usually,  a 
standard  intervention  is  tested  against  a  slightly  modified  version. 

C.  Outcome  Evaluation 

Outcome  evaluation  assesses  how  well  an  AIDS  activity  or  project  (the  outcome 
objective  stated  in  the  workplan)  was  carried  out.  There  is  a  common  misperception  that  in 
order  for  community-based  organizations  to  do  an  outcome  evaluation  of  HIV/STD  projects, 
they  need  to  show  how  and  by  how  much  behaviors  of  targeted  groups  changed  as  a  result  of 
the  project.  Actually,  community-based  organizations  can  conduct  outcome  evaluations 
without  having  to  demonstrate  direct  behavior  changes  of  target  groups.  For  example, 
community-based  organizations  can  measure  how  well  target  groups  learned  information 
taught  during  a  series  of  AIDS  education  seminars  through  such  evaluation  techniques  as 
surveys,  role  plays,  and  direct  interviews  with  clients;  they  can  seldom  measure  whether 
target  groups  are  actually  putting  into  daily  practice  what  they  have  learned. 

Outcome  evaluation  measures  relatively  short-term  program  results  or  benefits  and 
determines  "what  is  different"  as  a  result  of  an  intervention  program.  It  determines  the 
extent  to  which  the  outcome  objectives  of  an  intervention  —  those  which  were  to  be  achieved 
during  the  year  or  during  the  project  period  —  were  in  fact  met,  and  also  documents  the 
effects  of  the  intervention.8  In  an  HIV/STD-related  education  or  prevention  program, 
outcome  evaluation  answers  such  questions  as  the  following: 

•  What  are  the  immediate  effects  of  the  program  on  clients?  What  results  are 
pre-  and  post-tests  showing  in  terms  of  education  efforts’  success  in 
eliminating  myths  and  providing  new  knowledge?  Can  participants 
demonstrate  new  knowledge,  such  as  naming  the  major  methods  of  HIV 
transmission?  Can  they  demonstrate  new  skills,  such  as  use  of  a  condom? 

Can  women  role  play  appropriate  methods  for  negotiating  safer  sex?  Can 
injecting  drug  users  demonstrate  how  to  use  bleach  and  water  to  clean  drug 
"works"? 
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•  Do  clients  indicate  an  intention  to  change  their  behavior  on  their  post-test 
surveys? 

•  Based  on  short-term  measures  or  limited  follow-up  with  clients  —  using 
informal  surveys,  conducting  direct  client  interviews,  or  counting  how  many 
calls  for  additional  HIV/STD  information  were  received  —  to  what  extent  did 
the  target  group  put  the  knowledge  gained  from  a  program  intervention  into 
practice? 

•  Do  community  measures  indicate  that  clients  are  taking  action  based  on  what 
they  learned  in  the  intervention?  Do  HIV  testing  facilities  report  an  increase 
in  demand  following  your  interventions,  and  when  asked,  do  people  requesting 
testing  report  that  your  program  led  them  to  come  for  testing?  Are  people 
requesting  more  information  about  HIV/AIDS  following  a  PSA  campaign 
suggesting  that  they  call  your  agency  for  such  information?  Are  agencies 
which  provide  bleach  reporting  increased  demand?  Are  condom  sales  in  your 
target  community  up? 

Many  approaches  used  for  design  and  process  evaluation  can  also  be  used  for  outcome 
evaluation.  These  include  pre-  and  post-tests,  post-session  assessments,  focus  groups,  and 
games  and  other  activities. 
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IV.  PROGRAM  EVALUATION  METHODS 


Evaluation  should  be  occurring  at  every  project  stage,  whether  in  the  design,  early 
implementation,  or  post-demonstration  stage.  Ideally,  evaluation  should  begin  early  in  the 
project  design  phase  and  continue  throughout  the  project’s  implementation.  The  aggregate 
data  collected  during  the  early  stages  are  necessary  for  facilitating  outcome  project  evaluation 
later. 


Carrying  out  such  evaluation  requires  not  only  understanding  basic  evaluation 
terminology  and  the  components  of  an  evaluation  system,  but  also  being  able  to  use  a  variety 
of  evaluation  techniques.  The  task  is  to  construct  an  appropriate  evaluation  system  for  your 
program  by  using  a  number  of  available  "building  blocks"  —  evaluation  techniques  which 
can  each  generate  particular  types  of  needed  information,  and  when  put  together  can  provide 
a  comprehensive  and  practical  evaluation  of  a  particular  HIV/STD  education  and  prevention 
program. 

The  descriptions  below  are  designed  to  introduce  a  number  of  evaluation  techniques 
which  are  likely  to  be  useful  in  evaluating  HIV/STD  education  and  prevention  programs. 

The  purpose,  potential  applications,  and  steps  or  procedures  for  using  each  technique  are 
explained.  Before  you  begin  to  construct  an  evaluation  plan  for  your  program,  you  should 
become  familiar  with  these  techniques. 

A.  Needs  Assessments 

A  needs  assessment  surveys  a  specific  population  in  order  to  gather  information  about 
community  needs  and  service/resource  availability,  and  identify  the  services/resources  that 
are  not  being  provided.  The  data  generated  from  a  needs  assessment  can  substantiate  already 
known  problems  and/or  identify  new  problems  within  a  population  or  community.  One 
should  conduct  a  needs  assessment  to  "test"  the  conditions  before  making  major  changes  or 
launching  entirely  new  services  in  an  agency,  neighborhood,  or  larger  community.  The 
structure  of  a  needs  assessment  can  vary,  depending  on  the  organization’s  size  and  resources. 
Because  many  Hispanic  community-based  organizations  do  not  have  sufficient  staff  or  funds 
to  conduct  formal  needs  assessments,  they  often  use  informal  methods.  The  following 
describes  both  types  of  needs  assessments. 

A  formal  needs  assessment  is  usually  a  written  survey  of  questions  disseminated  to  a 
particular  population  —  often  a  combination  of  residents,  community-based  organization 
representatives,  and  community  leaders  —  for  their  responses.  An  informal  needs 
assessment  can  be  oral,  using  a  group  leader  to  ask  pre-planned  questions  to  an  organized 
group,  with  an  observer  present  to  take  notes  on  responses.  It  can  even  be  more  informal, 
where  people  from  the  community  just  tell  staff  what  services  are  needed,  or  staff  themselves 
determine  needs  based  on  their  observations  of  clients  and  programs.  When  using  informal 
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methods,  always  document  the  information,  especially  such  information  as  the  date,  source  of 
information  (e.g.,  staff,  clients,  etc.),  and  description  of  needs,  otherwise  this  information 
will  be  lost.  Then,  review  and  discuss  this  information  at  staff  meetings  and  make  sure  to 
record  any  actions  taken. 

A  needs  assessment  is  most  effective  when  the  data  are  compiled,  analyzed, 
documented  in  written  form,  and  reviewed  by  key  staff.  The  final  report  serves  as  a  guide, 
indicating  what  services  need  to  be  changed  or  added  to  an  existing  program,  or  indicating 
the  need  for  a  new  program.  The  following  four  points  can  help  you  design  and  implement  a 
community  needs  assessment: 

1.  Identify  the  questions  to  be  answered.  These  should  include  some  of  the  following: 

•  Who  lives  in  the  community?  What  are  the  socioeconomic  characteristics  of 
the  residents?  What  are  the  sizes  and  characteristics  of  special-needs 
populations,  such  as  youth,  single  mothers,  and  the  homeless? 

•  What  is  the  extent,  location,  and  accessibility  of  health  and  AIDS  education 
and  prevention  programs  within  the  community?  What  critical  health-related 
needs  are  not  being  met  and  what  are  the  effects  of  these  service  gaps  on  the 
community  and  its  residents? 

•  What  are  the  perceived  major  problems  within  the  community,  and  to  what 
extent  is  there  consensus  about  the  relative  importance  of  these  problems? 

2.  Review  available  information,  and  assess  other  data  needs.  In  order  to  answer 
these  and  other  questions,  many  different  types  of  information  should  be  obtained  and 
reviewed,  through  extensive  use  of  secondary  sources,  and  perhaps  some  individual 
interviews  with  agency  directors  and  other  community  experts: 

•  Information  from  any  previously  conducted  needs  assessments,  whether  done 
for  the  whole  community  or  for  a  particular  organization,  entity,  or 
subdivision  of  the  community. 

•  Federal,  state,  and  local  health  department  AIDS,  STD,  and  other  disease 
surveillance  data. 

•  Information  about  the  economic  base  of  the  community,  including  business 
establishments  and  their  employment  profiles  and  products/ services. 

3.  Conduct  interviews  with  community  residents  and  leaders.  Given  the  limited 
resources  of  most  community  needs  assessments,  the  most  appropriate  approach  to 
conducting  resident  and  community  leader  interviews  is  often  to  admit  that 
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"generalizable"  data  based  on  a  random  sample  probably  cannot  be  obtained  with  the 
resources  available,  and  then  seek  to  obtain  broadly  representative  case  study  data 
instead.  This  can  be  done  in  a  variety  of  ways.  The  following  approach  has  been 
successful  in  a  number  of  locales: 

•  Decide  what  information  you  need  that  should  be  obtained  through  individual 
or  group  interviews  or  "focus  groups." 

•  Develop  a  questionnaire  which  is  reasonable  in  length,  and  contains  primarily 
closed-ended  questions  (these  questions  elicit  either  "yes"  or  "no"  answers 
which  are  much  easier  to  tabulate)  but  also  opportunities  for  comments  (needed 
for  depth  of  understanding). 

•  Identify  major  categories  of  individuals  who  should  be  interviewed  —  residents 
of  varying  age,  sex,  or  socioeconomic  status;  businesspersons;  human  service 
agency  directors  and  staff;  public  officials;  religious  leaders;  educators;  other 
identifiable  groups. 

•  Decide  how  many  interviews  you  can  afford  to  conduct  and  allocate  them 
among  the  various  categories  of  individuals  identified  above.  For  example,  for 
the  residents  subgroup,  look  back  at  the  profile  you  developed  of  community 
residents,  and  develop  a  plan  for  interviewing  people  of  varying 
characteristics.  You  might  decide  to  interview  50  residents,  allocated  to  fit 
community  profiles  based  on  age,  race/ethnicity,  income,  etc.  This  is  your 
sampling  plan.  The  resulting  data  will  not  be  statistically  representative  of  the 
whole  community,  but  you  will  get  valuable  case-study  data. 

•  Test  the  questionnaire  with  several  people  of  differing  categories  —  and  then 
revise  any  unclear  questions  and  shorten  if  necessary. 

•  Train  interviewers.  For  interviews  with  Hispanic  residents,  interviewers 
should  generally  be  bilingual.  Make  sure  the  interviewers  fully  understand  the 
questions  and  also  understand  how  to  carry  out  interviews  and  record  data 
objectively,  consistently,  and  carefully. 

•  Go  to  agencies,  churches,  and  other  entities  and  ask  them  to  identify 
individuals  with  specified  characteristics  (based  on  your  sampling  plan)  for  you 
to  interview. 

•  Carry  out  these  interviews.  Also  interview  the  businesspeople,  public 
officials,  and  other  categories  of  individuals  you  probably  identified. 
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•  Now  tabulate  the  data  separately  by  type  of  person  interviewed  (e.g., 
community  leaders  versus  residents),  so  you  can  compare  responses  by  groups. 
Look  for  areas  of  consensus  or  disagreement  concerning  community  problems 
and  needs. 

•  Go  back  to  several  agencies  and  ask  them  to  help  you  get  together  groups  of 
people  of  various  characteristics  —  a  group  of  6-10  youth,  women  with  small 
children,  etc.  —  and  have  a  group  discussion  to  further  explore  some  of  the 
problems  and  solutions  suggested  in  the  individual  interviews. 

4.  Analyze  the  data  and  prepare  a  report.  Once  you  have  completed  primary  data 

collection  as  well  as  obtained  available  existing  secondary  data,  you  are  ready  to  look 
over  all  the  information,  look  back  at  the  questions  you  wanted  to  answer,  identify 
and  fill  any  data  gaps,  and  start  reviewing  your  data.  Organize  the  information  by 
issue,  in  order  to  answer  the  major  study  questions.  Use  the  following  format  as  a 
guide  for  organizing  the  report: 

•  Provide  background  on  how  the  study  was  carried  out,  questions  it  was 
designed  to  answer,  and  any  limitations  of  your  approach. 

•  Provide  a  community  profile  —  physical  characteristics,  facilities  and  services, 
and  residents. 

•  Identify  major  community  strengths  and  problems,  and  present  areas  of 
consensus  and  agreement,  as  well  as  disagreement,  among  residents  and 
leaders  concerning  the  nature  and  priority  of  problems  and  methods  for  their 
resolution. 

•  Present  detailed  information  on  areas  of  special  concern,  such  as  HIV/STD 
education  and  outreach  services.  Deal  with  available  resources,  needs  and 
gaps,  and  community  recommendations  for  problem  solving. 

•  Provide  conclusions  which  identify  the  implications  and  importance  of  your 
findings,  and  provide  recommendations  for  action  to  solve  the  identified 
problems. 

B.  Expert  Reviews 

Consult  regularly  with  experts  outside  your  agency  to  assure  accuracy  and 
effectiveness  of  information  to  be  presented  to  the  community.  All  written  and  visual 
HIV/STD-related  materials  designed  to  educate  Hispanic  community  groups  or  promote  a 
specific  project  or  workshop  should  undergo  review  by  Hispanic  experts  working  in  the  field. 
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For  example,  if  your  agency 
plans  to  offer  support  groups  to 
Hispanic  families,  consult 
persons  who  are  experienced  in 
organizing  support  groups  for 
Hispanics  and  understand  the 
obstacles  to  family  participation 
in  support  groups.  If  your  client 
group  includes  injecting  drug 
users,  identify  persons 
experienced  in  working  with  this 
group;  if  your  program  is 
focusing  on  Hispanic  youth  as 
peer  AIDS  educators,  then 
contact  persons  knowledgeable 
about  this  role  for  youth. 

Outside  experts  can  also  assist 
project  planners  by  reviewing  English-  and  Spanish-language  materials  for  content,  language 
appropriateness,  and  accuracy. 

Use  experts  as  part  of  a  group  or  individually.  If  time  constraints  hamper 
participation  in  formal  group  meetings,  individual  experts  can  provide  a  variety  of  assistance 
at  their  convenience,  such  as  reviewing  proposals,  materials,  and  intervention  strategies. 
Using  more  than  one  expert  can  provide  a  range  of  perspectives.  For  example,  experts  can 
participate  in  a  Delphi  process,  in  which  each  person  provides  comments  or  written  revisions 
to  a  project  proposal  or  educational  materials,  then  changes  are  made  and  the  process  is 
repeated,  sometimes  through  several  cycles,  until  the  product  is  largely  acceptable  to  the 
entire  group. 

An  expert  should  have  sufficient  knowledge  of  and  field  and/or  personal  experience  in 
the  subject  or  in  a  closely-related  subject.  Expertise  may  be  based  on  formal  education  or 
experience.  Former  and  recovering  substance  abusers,  persons  living  with  HIV/ AIDS,  and 
parents  of  persons  with  AIDS  can  offer  assistance  and  input  on  key  issues  to  be  covered  in  a 
case  management  or  HIV  support  group  program.  When  gathering  a  pool  of  experts, 
consider  using  not  only  researchers  and  heads  of  HIV/STD  programs,  but  also  social 
workers,  teachers,  STD  clinic  nurses,  counselors,  and  other  experts  knowledgeable  about 
specific  populations  (e.g.,  farmworkers,  Hispanic  subgroups)  in  order  to  obtain  other 
perspectives.  Be  selective  about  whom  you  choose  as  experts  for  your  project,  always 
keeping  in  mind  the  type  of  input  you  want  to  obtain. 

There  are  several  excellent  sources  for  identifying  and  locating  experts.  A  good 
starting  point  for  finding  local  experts  is  other  nonprofit  organizations  in  the  same 
community.  Other  organizations  may  either  have  someone  with  the  relevant  expertise  willing 
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WAYS  TO  USE  EXPERTS  TO  EVALUATE 
PROGRAM  DESIGN 

•  To  serve  on  a  panel  which  meets 
periodically  to  review  project  progress  and 
interventions,  and  offer  expert  advice. 

•  To  play  a  more  active  consultant  role  by 
developing  interventions,  designing  flyers, 
games,  role  plays,  or  materials. 

•  To  serve  as  problem-solvers  when  critical 
needs  arise. 


to  provide  assistance  or  be  able  to  direct  you  to  other  community  experts  outside  the  agency. 
Local  or  state  health  departments,  high  schools,  colleges,  and  universities  should  also  be 
contacted.  For  very  specific  issue  areas,  such  as  methadone  treatment,  you  may  need  to  first 
identify  agencies  working  directly  or  indirectly  in  this  area,  such  as  a  local  health 
department’s  substance  abuse  unit. 


In  addition  to  contacting 
local  and  state  agencies,  you 
should  also  contact  national 
organizations  such  as  the 
National  Council  of  La  Raza 
(NCLR),  which  works  closely 
with  affiliate  organizations  all 
over  the  country.  (NCLR  is 
currently  developing  a  directory 
of  experts  in  various  health  and 
AIDS-related  areas.)  Be  aware 
that  it  may  take  numerous 
telephone  calls  before  you  find 
the  expert(s)  that  fit  your  specific 
project  needs. 

When  contacting  potential  experts,  it  is  important  to  clearly  identify  the  tasks 
involved,  the  type  of  expertise  needed,  the  timeline  for  completing  the  task,  and  the  projected 
amount  of  expert  time  required.  Offer  background  information  about  the  nature  of  your 
project  and  about  your  organization.  Be  prepared  to  pay  fees  for  expert  services. 

C.  Community  Groups 

Always  seek  input  from  community  groups  (e.g.,  local  residents,  clients  and 
volunteers  of  community-based  organizations,  and  staff  of  other  local  agencies)  when 
designing  or  revising  an  AIDS  education  and  prevention  project  for  the  community.  They 
know  best  the  unique  concerns  and  problems  of  community  residents,  and  kinds  of  services 
needed  in  the  community.  These  groups  can  provide  valuable  suggestions  about  cultural 
issues  to  be  aware  of  while  conducting  effective  outreach  in  the  community.  Using  feedback 
from  groups  can  enhance  the  success  of  your  program. 

Located  in  the  heart  of  the  community,  community-based  organizations  have  the 
advantage  of  accessibility  to  many  of  the  individuals  that  HIV/STD  programs  target.  The 
challenge  for  the  staff  of  community-based  organizations  lies  not  in  finding  individuals  to 
serve  as  members  of  a  group,  but  in  organizing  these  individuals  and  groups  in  ways  that 
will  assist  staff  in  designing  and  implementing  the  most  appropriate  programs  for  the 


SOURCES  FOR  FINDING  EXPERTS 

•  Nonprofit  organizations  in  your  community 

•  Local  or  state  health  departments 

•  High  schools,  colleges,  and  universities 

•  National  organizations  (e.g.,  NCLR) 
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community  or  for  assessing  the  quality  of 
materials  or  training  sessions.  Community 
groups  can  provide  input  in  the  design  of  a  teen 
peer  AIDS  educator  program  or  an  ongoing 
street  outreach  program  targeting  injecting  drug 
users.  Focus  groups,  community  interviews, 
and  demonstration  or  pilot  sessions  are  three 
approaches  for  obtaining  feedback  from 
community  groups.  Careful  staff  consideration 
of  the  results  of  these  approaches  can  enhance 
the  project’s  responsiveness  to  the  needs  of  the 
target  groups.  Although  the  results  of  these  various  approaches  are  informative,  they  cannot 
be  generalized  to  a  broader  population. 

1.  Focus  Groups 

A  focus  group  is  a 
carefully  planned  discussion 
among  a  small  group  of  perhaps 
8-12  people  led  by  a  trained 
moderator.  The  purpose  of  a 
focus  group  is  to  generate 
reactions  to  and  opinions  about 
concepts,  approaches,  and 
materials,  including  PSAs, 
posters,  newspaper  or  magazine 
risk-reduction  advertisements, 
flyers  or  brochures,  outreach 
methods,  or  risk-reduction 
training  approaches.  The 
facilitator  or  moderator  should  be 
someone  who  is  not  involved  in 
the  design  or  revision  of  the 
brochure  or  project.  This 
reduces  bias  that  can  result  from 
opinions  expressed  by  persons 
working  directly  with,  or  having 
a  vested  interest  in,  the  project. 

The  focus  group  should  consist 
of  individuals  from  the  desired 
target  population  having  certain 
common  characteristics  relevant 

to  the  topic  involved.  The  more  your  focus  group  can  relate  to  the  topic  or  problem  being 


IMPORTANT  STEPS  FOR 
ORGANIZING  A  FOCUS  GROUP 

•  Specify  the  purpose  for  organizing  a  focus 
group. 

•  Prepare  a  written  outline  or  a  guide  for  use 
by  the  moderator  to  assure  that  certain 
desired  topics  and  questions  are  addressed 
during  the  focus  group  discussion. 

•  Develop  a  strategy  for  selecting  focus  group 
participants  who  are  representative  of  the 
target  group,  and  who  have  not  seen  the 
exact  materials,  workshop,  or  PSA  before. 

•  Recruit  a  trained  facilitator  who  is  familiar 
with  focus  groups  and  is  not  involved  in  the 
project. 

•  Be  aware  of  limited  language  proficiency  or 
literacy  among  focus  group  participants. 

•  Plan  to  have  at  least  two  observers  to  take 
notes  of  comments  and  reactions  during  the 
session. 


WAYS  TO  USE  COMMUNITY 
GROUPS  FOR  EVALUATION 

•  Focus  Groups 

•  Community  Interviews 

•  Demonstration  Sessions 
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targeted,  the  more  valuable  the  information  will  be.  For  example,  to  test  the  approaches  or 
strategies  for  carrying  out  an  AIDS  education  program  for  youth  in  a  specific  area,  organize 
a  focus  group  consisting  of  youth  from  the  community  you  are  targeting. 

Focus  groups  provide  a  relatively  low-cost  way  to  test  HIV/STD  education  and  risk 
reduction  concepts  or  materials  before  they  are  introduced  to  the  public.  A  technique 
developed  for  marketing  research  —  to  test  new  products,  packaging,  advertising  slogans, 
and  approaches  —  its  results  are  less  precise  than  a  large-scale  survey,  but  it  is  also  far  less 
expensive  and  complicated  to  conduct.  It  can  help  a  prevention  program  identify  the 
strengths  of  and  potential  problems  with  its  prevention  efforts  before  committing  itself  to 
large-scale  production  or  dissemination.  For  example,  using  a  focus  group  to  review  a 
proposed  PSA  might  result  in  a  determination  that  the  message  is  not  clear  —  and  will  enable 
the  project  to  revise  it  before  sending  out  half  a  dozen  tapes  to  local  radio  stations  for 
widespread  broadcasting.  A  national  media  campaign  might  use  test  marketing  in  one  or 
several  cities  to  test  its  PSAs;  a  community-based  organization  takes  similar  precautions 
when  it  tests  a  local  PSA  in  a  focus  group.  Similarly,  if  several  different  messages  are  being 
considered  for  a  local  media  campaign,  a  focus  group  can  be  used  to  see  how  the  proposed 
target  group  reacts  to  each,  and  suggest  which  approach  seems  more  promising.  The 
following  case  study  illustrates  the  use  of  focus  groups. 


CASE  STUDY 

Using  a  focus  group  to  conduct  design  evaluation 

Staff  of  the  "Positive  Teen"  project  have  been  asked  by  the  state  health  department 
to  preview  and  provide  comments  to  a  bilingual  video  documentary  which  will  be 
finalized  and  released  in  the  next  six  months.  The  video  is  on  HIV/STDs  and  is 
designed  to  educate  Hispanic  high  school  teenagers  who  are  primarily  English 
proficient  and  also  speak  and  understand  Spanish.  Since  a  major  activity  of 
Positive  Teen  includes  conducting  presentations  on  HIV/STDs  for  high  school 
teenagers,  the  AIDS  educators  are  always  interested  in  finding  new  materials  and 
videotapes  to  use  in  their  sessions.  Because  the  video  is  aimed  at  Hispanic 
teenagers,  the  staff  have  decided  not  just  to  preview  it  themselves,  but  also  to 
organize  a  focus  group  of  teenagers  to  react  to  it. 


In  planning  and  organizing  the  focus  group,  staff  need  to  consider  and  be  able  to 
answer  the  following  questions  (answers  are  based  on  the  above  case  study): 
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What  is  the  purpose  for  organizing  a  focus  group  ? 

Positive  Teen  staff  wish  to  get  comments  and  reactions  on  a  pilot  videotape 
from  Hispanic  teenagers,  who  are  the  target  audience  for  the  videotape. 

What  will  be  presented  to  the  focus  group? 

A  video  documentary  on  HIV/STDs. 

Who  will  moderate  /facilitate  the  focus  group? 

Positive  Teen  staff  have  had  a  long-standing  working  relationship  with  a  local 
community-based  agency.  The  agency’s  project  director  and  health  services 
coordinator  have  agreed  to  assist  Positive  Teen  staff  since  they  have  led  focus 
groups  in  the  past.  (In  this  case,  Positive  Teen  staff  could  moderate  the  focus 
group  themselves  since  they  are  not  directly  involved  in  the  design  of  the 
videotape,  but  they  have  never  used  a  focus  group  before,  so  prefer  to  learn  by 
observing  the  process.) 

How  will  focus  group  participants  be  selected? 

Positive  Teen  staff  work  closely  with  the  guidance  counselors  in  local  high 
schools.  A  high  school  guidance  counselor  who  is  Hispanic  and  knows  the 
Hispanic  students  well  will  recruit  ten  Hispanic  high  school  students  —  equal 
numbers  of  boys  and  girls  evenly  divided  between  juniors  and  seniors  —  to 
participate  in  the  focus  group. 

To  what  extent  have  these  individuals  been  exposed  to  HIV/ STD-related  education  and 
materials? 

The  ten  students  have  received  HIV/STD  education  by  Positive  Teen  staff,  but 
will  be  screened  to  be  sure  that  they  have  not  seen  any  video  documentaries  on 
the  subject. 

How  will  you  assess  whether  the  participants  in  the  group  have  limited  English 
language  or  literacy  abilities? 

Prior  to  the  session,  students  will  be  asked  individually  which  language  they 
are  most  proficient  in  for  the  educational  session.  Four  out  of  ten  students 
indicated  proficiency  in  both  Spanish  and  English.  Staff  will  also  ask 
questions  to  assess  literacy  in  both  languages  (e.g.,  Do  you  feel  most 
comfortable  reading  (and  writing)  in  English  or  in  Spanish?  What  kinds  of 
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materials  do  you  enjoy  reading  the  most  —  newspapers,  magazines,  novels, 
etc.?  How  often  do  you  read?)  Answers  to  these  questions  suggested  to  staff 
that  students  were  most  literate  in  English. 

If  there  are  participants  with  limited  English  language  proficiency  or  literacy,  how 
will  these  be  addressed  during  the  focus  group? 

Since  there  will  be  four  Spanish  speakers  in  the  focus  group,  the  focus  group 
will  be  entirely  bilingual.  Two  bilingual  moderators  will  lead  the  group,  and 
will  trade  off  translating  the  questions  to  the  group. 

Other  than  comments  from  the  group,  will  any  other  methods  be  used  to  assess  the 
videotape  ? 

Before  the  session,  students  will  be  asked  to  complete  a  short  knowledge, 
attitudes,  beliefs,  and  behaviors  (KABB)  survey  which  will  be  available  in 
English  and  Spanish;  staff  will  ensure  that  the  survey  requires  not  more  than  a 
fifth  grade  reading  level.  A  post-session  survey  containing  the  same  questions 
will  be  administered  to  students  immediately  following  the  videotape. 

Who  should  be  present  at  the  focus  group  session  ? 

The  ten  students:  focus  group  participants 
Positive  Teen  staff:  observer,  notetaker 

Local  community-based  agency  health  staff:  two  bilingual  moderators 

The  moderators  should  guide  the  discussion  using  a  predetermined  outline  of 
questions,  as  shown  on  the  next  page.  The  focus  group  should  last  for  a  predetermined 
amount  of  time,  with  the  moderators  keeping  the  discussion  on  the  topic.  Once  the 
moderators  feel  all  the  important  points  have  been  covered,  they  should  conclude  the  session 
and  thank  all  the  participants.  The  moderators,  observers,  and  staff  of  Positive  Teen  should 
schedule  a  meeting  soon  after  to  discuss  the  results  of  the  focus  group  session.  This  should 
be  followed  by  a  written  report  which  should  guide  the  use  of  findings.  (See  page  30  for 
characteristics  of  a  successful  focus  group.) 
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SAMPLE  QUESTIONS  FOR  THE  FOCUS  GROUP 

•  What  part(s)  of  the  video  did  you  enjoy  most/least?  Why? 

•  What  did  you  think  about  the  length  of  the  videotape?  Was  it  too  short,  too 

long,  about  right? 

•  What  did  you  think  about  the  characters  in  the  videotape?  How  realistic  were 
the  characters  in  this  documentary? 

•  What  about  the  language  used  in  the  videotape?  How  easy  or  hard  was  it  to 
understand?  What  parts  were  unclear  or  confusing? 

•  How  easy  or  hard  was  it  to  understand  the  commentator?  What  did  you  think 
about  his/her  speaking  pace  and  tone  of  voice?  How  convincing  were  the 
commentator’s  messages? 

•  What  did  you  learn  from  watching  this  video? 

•  How  did  the  video  presentation  change  your  feelings,  attitudes,  and/or  opinions 

about  HIV/AIDS  and  persons  infected  with  HIV? 

•  What  does  it  mean  to  "practice  safer  sex"? 

•  How  do  you  think  sexually  active  teenagers  would  react  to  adopting  safer  sex 
practices  (if  they  don’t  already)  after  watching  this  videotape? 

•  What  questions  did  the  video  leave  you  with?  Should  any  particular  questions 
be  addressed  in  a  group  discussion  following  the  video? 

•  How  would  you  feel  about  viewing  this  video  as  a  part  of  an  AIDS  101  training 
seminar?  Why  do  you  feel  that  way? 

•  What  kind  of  discussion  do  you  think  is  needed  along  with  the  video,  so  its 
message  will  get  across  to  teenagers? 
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CHARACTERISTICS  OF  A  SUCCESSFUL  FOCUS  GROUP 

•  The  objective  of  the  focus  group  is  clearly  defined  and  manageable  --  to  decide 
among  several  different  HIV/STD  outreach  strategies  for  runaway  youth,  to 
determine  the  clarity  and  attractiveness  of  a  planned  brochure  or  flyer. 

•  The  moderator  is  experienced  at  facilitating  group  discussions  and  is  independent  of 
the  project  so  that  s/he  is  knowledgeable  but  does  not  bias  the  discussion  with  his/her 
own  views. 

•  The  discussion  outline  provides  a  blueprint  for  the  focus  group  by  stating  the  scope 
of  the  focus  groups  and  the  major  questions  or  issues  to  be  covered. 

•  Key  program  staff  either  observe  or  listen  from  another  room. 

•  The  focus  group  environment  is  well-selected  —  it  is  comfortable,  quiet,  and  allows 
for  good-quality  audio  taping;  refreshments  are  served. 

•  Audio-visual  equipment  works  properly. 

•  The  group  composition  represents  the  target  group  of  the  project;  if  there  are  several 
target  groups,  multiple  focus  groups  are  used  so  each  group  is  as  homogeneous  as 
possible. 

•  The  group  is  a  manageable  size,  large  enough  to  provide  for  a  variety  of  views  but 
small  enough  that  everyone  participates  in  the  discussion;  usually  8-12  participants. 

•  The  session  is  carefully  scheduled  and  managed.  Usual  stages  include  an 
introduction,  a  warm-up  to  introduce  group  members,  general  topic  presentation  and 
discussion  to  obtain  broad  opinions  about  HIV/STD  prevention;  presentation  of  the 
concept,  and  specific  discussion  of  reactions,  with  both  individual  opinions  and 
group  discussion;  a  closing  and  a  thank  you. 

•  The  session  is  of  an  appropriate  length,  one  and  a  half  to  two  hours  maximum. 

•  Focus  group  participants  receive  a  reasonable  fee  for  their  participation  (usually  $25 
or  $30;  an  extra  $5  might  be  added  for  an  evening  session). 

•  Discussion  among  program  staff,  moderator(s),  and  observer(s)  follows  each  focus 
group. 

•  Once  all  focus  groups  have  been  completed,  a  comprehensive  report  is  prepared. 
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2. 


Community  Interviews 


A  community  interview  is  a  less  formal  version  of  the  focus  group.  This 
technique  involves  a  similar  discussion  approach,  but  participation  is  relatively  open.  Where 
a  focus  group  involves  the  actual  target  group,  the  community  interview  involves  a  mixed 
group  of  individuals  —  often  all  ages,  genders,  sexual  orientations,  and  racial/ethnic 
backgrounds.  For  instance,  a  community  interview  may  involve  a  group  of  clients  of  a 
particular  health/human  service  program,  all  students  in  a  class,  or  a  group  of  parents  of 
Head  Start  children  in  a  particular  center.  Because  of  the  greater  diversity  of  individuals,  a 
community  interview  can  involve  a  larger  group  than  a  focus  group.  A  good-sized  group  is 
anywhere  between  15  and  30  individuals. 

Similar  to  the  focus  group,  a  community  interview  involves  a  well-structured 
discussion  session.  At  least  two  group  leaders  facilitate  the  discussion  using  a  carefully 
developed  interview  guide.  The  community  interview  should  be  planned  and  conducted  with 
care,  and  its  proceedings  should  be  both  taped  and  recorded  in  writing.  Because  a 
community  interview  is  informal,  its  results  should  be  viewed  as  providing  valuable  insight 
but  not  information  which  can  be  projected  to  a  broader  population.  Community  interviews 
are  recommended  for  assessing  PSAs,  videos,  television  documentaries,  posters,  games,  role 
plays,  skits,  and  all  other  written/visual  materials  and  methods  used  for  educational  purposes. 
The  materials  to  be  tested  can  be  directed  at  specific  population  groups  (e.g.,  Hispanic 
males,  teenagers,  etc.)  or  can  be  general  in  focus. 


CASE  STUDY 

Using  a  community  interview  to  preview  an  HIV/AIDS  skit 

Positive  Teen  staff  coordinated  a  group  of  community  volunteers  consisting  of 
teenagers,  high  school  teachers,  and  parents  to  preview  and  provide  feedback  to 
an  HIV/ AIDS  education  skit.  At  the  end  of  the  skit,  the  facilitator  —  a  local  high 
school  teacher  —  asked  the  group  the  following  questions  about  the  skit: 

•  What  is  this  skit  all  about? 

•  Which  character  roles  did  you  like  best/least? 

•  What  messages  did  you  get  from  this  skit? 

•  How  is  this  skit  useful  to  teenagers?  To  parents?  To  teachers? 

•  How  would  you  improve  the  skit  to  make  it  more  effective  in  getting  its 
message  across?  (See  Worksheet  B.) 
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3. 


Demonstration  Sessions 


A  demonstration  session  is  a  "practice  run"  or  a  "pilot  session"  of  the  actual 
session  you  are  planning  to  conduct.  It  is  the  opportunity  to  test  how  HIV/STD-related 
presentations  and  seminars  "go  over"  with  participants  who  represent  the  desired  target 
group,  before  the  sessions  are  widely  or  repeatedly  implemented.  Once  the  various 
components  of  the  seminar  —  agenda,  curriculum,  written  and/or  audio-visual  materials  — 
have  undergone  sufficient  review  using  one  or  more  assessment  techniques  described  above, 
you  should  do  a  demonstration  session.  The  results  of  the  session  may  indicate  a  need  for 
either  significant  redesign  or  minor  modifications  and  fine-tuning.  Even  when  you  have  not 
used  other  assessment  techniques  such  as  experts  and  focus  groups,  you  can  and  should  still 
conduct  a  demonstration  session  as  long  as  you  allow  ample  time  to  make  any  necessary 
changes.  Clearly,  the  benefits  will  be  greater  when  several  evaluation  techniques  are  used  on 
a  project. 

You  should  do  a  practice  run  of  all  HIV/STD-related  sessions.  These  could  include 
slide/video  presentations,  theatrical  presentations,  in-house  training  workshops,  and  other 
HIV/STD  presentations  for  use  in  schools,  beauty  parlors,  homes,  jails,  or  street  outreach 
sessions.  Because  a  demonstration  session  is  a  pilot  of  the  actual  session,  conduct  it  in  the 
same  way  you  plan  to  implement  the  ongoing  sessions.  Select  participants  who  fit  your 
target  audience;  use  the  person  who  will  actually  facilitate  the  seminar  or  workshop;  have 
ready  the  curriculum,  agenda,  and  other  materials  you  plan  to  use;  make  sure  to  hand  out 
any  surveys  and/or  forms  you  plan  to  use,  such  as  pre-  and  post-tests  and  assessments;  and 
when  possible,  conduct  the  session  in  the  actual  setting  where  the  activity  is  to  take  place. 
Plan  to  have  at  least  two  observers  present  who  do  not  take  part  in  the  discussion.  They 
should  be  knowledgeable  about  the  topic  but  not  necessarily  involved  in  the  specific  project 
that  is  being  tested.  Their  primary  role  is  to  take  notes  on  how  the  session  proceeds  and  any 
problems  that  arise  during  the  session.  Problems  that  involve  participants,  facilitators, 
materials,  and  even  the  location  should  be  documented  on  a  pre-developed  form  (non¬ 
participant  observation  form).  After  the  session,  assess  the  results  carefully  from  both  the 
participant  and  the  agency  perspective,  and  revise  the  seminar  based  on  that  assessment. 
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BEFORE  A  DEMONSTRATION  SESSION,  PREPARE 

•  Clear  objective(s)  for  the  proposed  activity  (e.g.,  by  the  end  of  the  AIDS 
education  session  for  adolescent  parents,  80%  of  the  participants  should  be 
able  to  identify  at  least  two  ways  to  reduce  the  risk  for  HIV  infection.) 

•  A  group  of  participants  representative  of  the  target  group. 

•  The  written  presentation  as  planned  for  implementation;  for  a  formal 
seminar,  this  means  a  full  seminar  package,  including  objectives,  agenda, 
group  exercises,  and  reference  materials. 

•  A  pre-  and  post-test  to  measure  knowledge,  attitudes,  and  behaviors 
(including  reported  plans  for  behavior  change)  --  either  written  or  oral, 
whichever  is  appropriate  for  the  group. 

•  Post-session  assessment  forms  or  a  list  of  questions  for  oral  interviews,  to 
determine  participant  reactions  to  the  presentation  methods  and  content. 

•  Non-participant  observation  forms  for  use  by  persons  knowledgeable  about 
HIV/ AIDS  but  not  directly  involved  in  the  project. 

•  A  detailed  session  assessment  form  for  use  by  the  facilitator  following  the 
demonstration  session. 


AFTER  THE  DEMONSTRATION  SESSION 

•  Have  a  detailed  post-session  staff  meeting  to  fully  assess  the  presentation. 

•  Aggregate  and  assess  the  results  of  the  session  based  on  pre-  and  post-tests, 
assessment  forms,  and  non-participant  observation  forms. 

•  Hold  a  debriefing  session  with  the  facilitator,  observers,  and  key  staff,  to 
review  results,  and  make  decisions  about  how  the  presentation  ought  to  be 
revised  based  on  the  demonstration  session. 

•  Make  appropriate  changes  in  the  presentation  or  seminar  —  in  terms  of 
objectives,  content,  agenda,  presentation  methods,  and  exercises  —  to 
overcome  problems  identified  during  the  demonstration  session. 
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D.  Observers 


An  observer  is  someone  who  is  present  at  a  presentation,  seminar,  or  other 
HIV/STD-related  educational  activity  involving  community  groups  or  individual  clients,  for 
the  purpose  of  recording  comments,  reactions,  criticisms,  or  problems  relevant  to  the 
activity.  This  person  can  provide  useful  insight  into  the  quality  of  a  presentation  or 
intervention,  in  terms  of  its  content,  clarity,  delivery,  and  appropriateness  for  the  target 
group. 


In  any  HIV/STD  intervention,  the  staff  or  volunteers  with  primary  implementation 
responsibility  —  presenter,  facilitator,  counselor,  or  media  specialist  —  are  likely  to  be  fully 
occupied  in  carrying  out  the  session  or  other  intervention.  Until  the  intervention  has  been 
repeated  several  times,  they  will  have  to  concentrate  on  that  process.  This  means  they  may 
not  be  fully  aware  of  the  reactions  of  participants  or  of  the  strengths  and  weaknesses  of  the 
presentations  of  others.  In  addition,  it  is  difficult  for  most  people  to  take  an  objective  look  at 
an  intervention  they  have  designed  and  are  implementing. 

A  carefully  selected  observer  can  be  helpful  in  providing  an  "outsider’s"  perspective. 
An  observer  may  be  a  participant  observer  who  is  a  member  of  the  target  group  but  is 
asked  to  pay  particular  attention  to  how  the  intervention  is  implemented,  or  a  non¬ 
participant  observer,  who  may  be  an  expert  on  HIV/STD  interventions  but  is  not  directly 
involved  in  the  presentation  or  project.  Non-participant  observers  can  be  persons  working  in 
other  areas  of  the  organization,  volunteers,  or  community  experts.  The  number  of  observers 
depends  on  the  size  of  the  group.  One  observer  should  suffice  for  a  group  of  ten,  and  two 
or  more  observers  may  be  necessary  for  a  group  exceeding  this  size. 

Observers  should  be  used  whenever  community  groups  are  brought  together  in  a 
specific  setting  to  provide  input  on  training  curricula,  brochures,  videotapes,  and  other 
materials.  They  should  also  be  used  during  process  evaluation  for  ongoing  HIV/STD 
education  and  prevention  activities.  For  instance,  observers  could  accompany  AIDS 
educators  periodically  when  they  conduct  outreach  activities  in  neighborhoods  and  schools,  in 
order  to  provide  the  educators  with  feedback  on  their  presentations.  The  following  example 
illustrates  one  possible  use  of  observers: 
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CASE  STUDY 

Using  observers  as  evaluators  during 
a  demonstration  session 

Staff  of  PARENTEEN  are  almost  ready  to  begin  implementing  an  HIV/STD 
education/skills-building  workshop  which  they  have  been  designing  for  the  last  six 
months.  In  preparation  for  a  demonstration  session  of  the  workshop,  staff  are 
recruiting  a  participant  and  a  non-participant  observer  to  help  evaluate  the  workshop 
before  its  full  implementation.  The  participant  observer  will  be  recruited  from  the  pool 
of  participants  who  have  already  agreed  to  attend  the  demonstration  session,  and  the 
non-participant  observer  will  be  one  of  the  agency’s  volunteers. 

As  part  of  the  recruitment  process,  an  informal  meeting  is  scheduled  between 
the  observers  and  a  PARENTEEN  staff  person  who  works  outside  of  the  HIV/AIDS 
component,  is  informed  of  the  proposed  workshop,  yet  has  had  no  involvement  in  its 
design.  During  the  meeting,  the  staff  person  provides  general  information  about  the 
demonstration  session  and  explains  the  functions  of  the  observers.  These  meetings 
provide  the  staff  with  a  good  indication  of  which  individuals  will  be  effective  observers 
for  the  demonstration  session.  PARENTEEN  staff  seek  individuals  who  meet  the 
following  criteria: 

•  Not  involved  in  the  project  or  presentation  to  be  given; 

•  Knowledgeable  about  the  topic  to  be  presented; 

•  Comfortable  with  notetaking; 

•  Familiar  with  and  attentive  to  group  dynamics;  and 

•  Impartial  during  the  session;  in  writing  notes,  the  person  is  as  objective 
as  possible  and  refrains  from  imposing  personal  viewpoints. 

Once  the  observers  have  been  recruited,  the  demonstration  session  of  the 
workshop  is  held.  The  observers  arrive  at  the  session  early  so  that  they  can  take  notes 
of  any  problems  before  the  session  begins  (e.g.,  any  missing  materials  or  equipment 
necessary  for  the  session,  unclear  directions  to  the  site,  etc.).  During  the  session,  the 
observers  take  notes  of  all  major  comments  and  questions  from  the  group  regarding  the 
written/visual  materials,  games  and  exercises,  and  level  of  group  participation.  After 
the  session,  PARENTEEN  staff  meet  with  the  observers  to  review  their  notes  and 
discuss  other  observations  about  the  workshop. 
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E.  Individual /Group  Assessment  Methods 

When  planning  training  sessions,  presentations,  or  workshops  for  groups,  plan  to 
include  some  methods  for  assessing  how  well  the  groups  learned  the  information  and  skills 
taught  during  the  session.  Identifying  the  areas  of  a  workshop  or  other  activity  that  need 
improvement  or  redesigning,  as  well  as  those  that  are  effective  can  benefit  the  entire  project 
and  its  target  groups.  Methods  for  obtaining  this  information  from  the  participants,  directly 
and  indirectly,  are  described  below;  these  include  written  assessments,  pre-  and  post-tests, 
surveys,  and  more  creative  approaches  such  as  games  and  other  activities,  case  studies  and 
role  plays,  and  key-person  interviews. 

1.  Post-Session  Assessments 


Post-session  assessments 
are  usually  one-  to  two-page  survey 
forms  which  are  handed  out  to 
participants  at  the  end  of  a  workshop  or 
training  seminar.  The  assessment  seeks 
to  determine  whether  participants 
learned  certain  information,  rejected 
stereotypes  or  myths,  or  changed 
attitudes  as  the  result  of  a  specific 
intervention.  This  assessment  tool  uses 
participants’  feedback  to  evaluate  the 
intervention.  Their  responses  provide 
some  indication  as  to  the  quality  and 
possible  effectiveness  of  the  intervention 
session. 


Every  workshop,  training 
seminar,  or  other  scheduled  and  formalized  prevention  presentation  should  include  some  form 
of  post-session  assessment.  Where  possible,  a  written  form  should  be  used.  Depending  on 
the  group’s  needs,  the  form  should  be  available  in  both  English  and  Spanish.  If  you  use  a 
written  form,  be  sure  to  indicate  that  anyone  who  would  prefer  to  be  interviewed  in  private 
has  the  option  of  requesting  this.  (This  is  one  way  to  deal  with  persons  who  have  limited 
literacy  without  embarrassing  them.)  Where  many  of  the  participants  are  unlikely  to  be  able 
to  complete  a  written  form,  then  either  all  or  a  sample  (every  third  participant,  perhaps) 
should  be  interviewed  orally.  Or,  the  participants  can  be  given  numbered  answer  sheets  so 
they  can  simply  circle  the  appropriate  numbered  response  to  a  series  of  closed-ended 
questions  asked  by  the  presenter  (e.g.,  response  number  one,  two,  or  three  to  a  particular 
question).  Entirely  oral  assessments  are  possible  if  the  group  leader  requests  that  participants 
raise  hands  to  respond  to  certain  questions,  and  a  sufficient  number  of  observers  are  present 
to  record  responses.  Regardless  of  the  method  used  to  administer  the  assessment,  the 
following  additional  considerations  should  guide  its  development  and  use: 


METHODS  FOR  ASSESSING 
INDIVIDUALS/GROUPS 

•  Post-Session  Assessments 

•  Pre-  and  Post-Tests 

•  Informal  Surveys 

•  Games  and  Other  Activities 

•  Case  Studies  and  Role  Plays 

•  Key-Person  Interviews 
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HINTS  FOR  DEVELOPING  AND  USING  POST-SESSION 

ASSESSMENTS 

•  Always  allow  sufficient  time  (at  least  fifteen  minutes)  at  the  end  of  the 
session  for  assessment. 

•  Keep  the  assessment  anonymous.  Many  individuals  may  not  fill  out  the 
form  or  may  not  give  honest  responses  if  their  names  are  attached  to  the 
assessment. 

•  Stress  in  the  session  the  importance  of  completing  the  assessment. 

•  Use  simple,  clear,  and  culturally  appropriate  language. 

•  Include  in  the  assessment  a  question  about  the  extent  to  which  participants 
have  learned  new  information,  skills,  or  attitudes  about  HIV/STDs. 

•  Include  a  question  on  how  participants  plan  to  apply  or  use  what  they  have 
learned.  Such  questions  are  "surrogate  measures"  for  actual  measurement 
of  behavior  change.  For  instance,  inquire  whether  participants  plan  to 
teach  others  about  HIV/STD  transmission;  or  whether  they  plan  to  change 
their  own  behavior  or  try  to  get  family  members  or  friends  to  change 
theirs. 

•  Use  mostly  closed-ended  questions,  so  the  form  can  be  completed  quickly. 

•  Include  some  open-ended  questions  so  that  participants  can  comment  on 
best  and  worst  aspects  of  the  session  (e.g.,  trainer,  presenter,  materials, 
etc.),  provide  other  comments,  or  make  recommendations  for  improving 
the  session.  (See  Worksheet  C.) 


2.  Pre-  and  Post-Tests 


Pre-  and  post-tests  are  usually  short  surveys  designed  primarily  to  assess 
changes  in  individuals’  responses  to  questions  regarding  HIV  and  other  STDs.  Pre-  and 
post-tests  follow  many  of  the  same  "rules"  as  post-session  assessment  forms,  but  they  are 
administered  twice.  Through  a  series  of  targeted  questions,  a  pre-test  measures  an 
individual’s  knowledge,  attitude,  and  sometimes  reported  behaviors  related  to  HIV/STD 
before  the  actual  intervention.  The  post-test,  given  after  the  intervention  has  taken  place, 
assesses  whether  an  individual  has  gained  new  knowledge  about  HIV/ AIDS  and  other  STDs, 
which  may  lead  to  changes  in  both  attitudes  about  the  disease  and/or  towards  people  infected 
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with  the  virus,  and  in  a  decision  to  change  sex  practices  and/or  drug-using  behavior.  The 
basis  of  measurement  is  the  change  in  response  between  the  pre-test  and  the  post-test. 

Results  of  pre-  and  post-tests  can  help  project  staff: 

•  Develop  a  general  understanding  of  a  group’s  baseline  knowledge  and  attitudes 
about  HIV/STD  transmission,  sexual  and  drug-using  practices,  and  prevention 
of  HIV/STDs. 

•  Identify  what  participants  learned  and  what  they  didn’t  learn,  and  whether  they 
plan  to  change  high-risk  behaviors. 

•  Determine  whether  the  presentation  works.  If  post-test  responses  show  little 
or  no  change  from  pre-test  responses,  then  it  is  important  to  try  to  identify  and 
resolve  the  problem(s)  for  future  sessions.  The  problems  may  be  related  to  the 
curriculum,  the  format,  the  presenters,  the  delivery  of  the  presentation,  etc. 

•  Make  revisions  for  the  next  intervention  session. 

Written  pre-  and  post-tests  may  not  be  appropriate  for  groups  who  are  not  regularly 
exposed  to  a  learning  environment  and  who  have  low  literacy  in  English  or  Spanish.  In  such 
cases,  pre-  and  post-tests  can  be  conducted  orally  in  small  group  settings,  or  other  creative 
methods  can  be  used  (e.g.,  use  of  index  cards  with  pictures,  role  plays,  etc.).  These  tests 
may  be  most  effective  for  groups  such  as  high  school  teenagers,  who  regularly  must  read  and 
write.  Groups  who  are  limited-English-proficient  and  literate  in  Spanish  should  be  provided 
Spanish  language  pre-  and  post-test  versions. 

Pre-  and  post-tests  offer  insight  as  to  whether  an  intervention  is  working.  These  tests 
cannot  establish  conclusively  that  it  was  the  intervention  that  led  to  any  recorded  changes  in 
knowledge,  attitudes,  or  planned  behaviors.  Such  conclusions  are  usually  based  on  findings 
from  using  a  case-control  group,  where  a  case  —  the  group  of  individuals  receiving  a 
particular  intervention  is  compared  to  a  control  group  —  the  group  of  individuals  not 
receiving  a  particular  intervention.  Without  a  control  group,  you  cannot  establish  that  it  was 
your  program  and  not  some  other  influence  (e.g.,  TV  program,  conversation  with  a  friend, 
etc.)  that  made  the  difference. 
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SEVEN  WAYS  TO  ENSURE  THE  EFFECTIVENESS  OF 
PRE-  AND  POST-TESTS 

1.  Check  all  questions  and  responses  with  several  experts  to  be  sure  that 
alternatives  are  appropriate  and  "correct"  responses  are  accurate. 

2.  Test  the  draft  forms  with  individuals  from  the  target  group,  to  be  sure  the 
questions  are  clear  and  understandable. 

3.  Code  each  set  of  pre-  and  post-test  forms  so  that  you  can  compare  the 
responses  of  the  same  individual  before  and  after  the  intervention. 

4.  Explain  the  importance  of  the  tests  at  the  beginning  of  the  session,  and  have 
the  pre-  and  post-tests  completed  individually.  If  you  are  using  a  written 
form,  make  sure  to  collect  the  pre-tests  so  they  are  not  changed  during  the 
session. 

5.  Use  primarily  closed-ended  questions.  If  you  want  to  minimize  guessing, 
give  instructions  that  ask  people  not  to  guess  if  they  really  don’t  know  the 
answer,  and  provide  a  "don’t  know"  response.  If  you  want  people  to  guess, 
don’t  give  them  this  alternative. 

6.  Make  sure  questions  asked  in  pre-  and  post-tests  directly  correspond  to  the 
information  presented  at  the  session.  This  is  important  to  consider  when 
borrowing  questions  from  other  programs. 

7.  Match  pre-  and  post-tests  for  individual  participants  and  tabulate  changes,  as 
well  as  tabulating  all  pre-  and  all  post-tests  and  comparing  them.  Then  use 
the  results  to  identify  areas  in  which  the  intervention  is  and  is  not  getting  the 
point  across  —  and  refine  the  presentation  or  seminar  to  strengthen  problem 
areas. 


3.  Informal  Surveys 

Informal  surveys  can  be  used  to  obtain  information  from  participants  in 
program  interventions  and  community  residents,  where  formal  KABB  studies  and  follow-up 
surveys  using  probability  samples  and  structured  questionnaires  are  not  feasible.  Results  of 
informal  surveys  should  give  some  indication  of  a  group’s  knowledge  but  cannot  be 
generalized  to  the  larger  population. 
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An  informal  survey  is  usually  administered  to  a  relatively  small  number  of  people  — 
perhaps  30  to  50.  Respondents  are  generally  selected  through  "quota"  sampling;  that  is,  the 
population  is  divided  into  target  groups  (e.g.,  injecting  drug  users,  teenagers,  women  of 
childbearing  age,  etc.),  and  a  pre-determined  number  of  respondents  are  chosen  from  each 
target  group  to  ensure  a  representative  sample  of  participants. 

To  obtain  baseline  data  on  knowledge  of  HIV/STDs  from  groups,  such  as  clients  of 
your  agency,  a  survey  could  be  provided  to  each  person  (or  every  third  person)  who  walks 
into  the  agency  on  a  given  day  or  over  a  one-week  period.  Interviewers  should  be 
responsible  for  handing  out  the  survey  after  the  person  has  gone  through  the  intake 
procedures.  The  interviewer  must  explain  the  purpose  and  importance  of  the  survey,  stress 
that  it  is  confidential,  and  request  the  participant  to  complete  it.  The  interviewer  should  be 
sensitive  to  persons  with  limited  English  proficiency  and  have  surveys  available  in  Spanish. 
Persons  with  limited  literacy  can  be  interviewed  individually  by  the  interviewer,  using  the 
questions  from  the  survey.  The  surveys  should  be  anonymous,  since  the  purpose  is  to  obtain 
aggregate  information.  In  order  to  identify  the  types  of  services  used  by  individuals, 

"purpose  of  visit"  should  be  one  of  the  questions  at  the  beginning  of  the  survey. 

Informal  surveys  can  also  provide  follow-up  information  on  people  who  participated 
in  a  particular  intervention.  Interviewers  can  work  with  agency  recordkeepers/intake  staff  to 
obtain  a  list  of  names  of  the  individuals  who  have  received  on-site  HIV/STD  education 
within  the  past  three  months.  A  note  or  symbol  of  some  kind  could  be  made  in  their  records 
indicating  that  these  individuals  should  receive  surveys  on  their  next  visit.  The  interviewer  is 
responsible  for  administering  the  survey  to  the  person,  who  can  complete  the  survey  while 
waiting  to  meet  with  a  staff  person. 

Community  or  "convenience"  sampling  is  another  option,  in  which  the  interviewers 
go  to  a  site  where  an  intervention  has  been  carried  out  —  drug  treatment  clinic,  STD  clinic, 
Headstart  Center,  or  community-based  organization  —  and  talk  to  people  who  come  to  the 
facility  and  have  been  involved  in  the  intervention.  As  a  matter  of  courtesy,  interviewers 
should  first  discuss  the  project  with  directors  of  the  respective  organizations  and/or  clinics 
and  seek  their  approval  and  input  on  how  best  to  conduct  interviews. 

While  results  will  not  be  statistically  significant  using  either  approach,  informal 
surveys  can  provide  valuable  information  about  whether  interventions  seem  to  be  making  a 
difference. 
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A  SUCCESSFUL  INFORMAL  SURVEY 

•  Uses  a  relatively  brief,  structured  survey  questionnaire  which  focuses  on  a 
limited  number  of  issues,  using  primarily  closed-ended  questions  but  also 
including  some  open-ended  ones. 

•  Is  implemented  by  a  small  number  of  well-trained  interviewers,  but  with 
more  flexibility  than  in  a  formal  survey;  so  long  as  they  obtain  responses 
to  the  key  questions,  they  may  add  open-ended  questions  to  follow  up  a 
response  or  clarify  an  issue. 

•  Is  administered  to  a  small  number  of  people  and  not  assumed  to  represent 
the  responses  of  a  larger  population. 

•  Usually  focuses  on  one  intervention  targeting  a  particular  group  rather  than 
a  whole  range  of  interventions  involving  a  wide  range  of  target  groups. 

•  Generates  qualitative  data  indicating  program  results,  strengths,  and 
weaknesses,  and  can  be  used  to  refine  the  intervention  approach  and 
document  promising  approaches. 


4.  Games  and  Other  Activities 


Creative  techniques  must  sometimes  be  used  to  assess  the  effects  of  a 
prevention  intervention.  When  participants  in  a  training  session  or  some  other  kind  of 
education  intervention  lack  basic  literacy  in  English  or  Spanish,  a  written  survey  in  either 
language  will  serve  little  use.  A  game  or  some  other  form  of  active  feedback  may  be  the 
best  method  to  gauge  the  extent  to  which  participants  have  learned  specific  information. 
Games  are  especially  valuable  because  they  require  active  involvement  of  the  group,  and  may 
require  participants  not  just  to  state  but  also  to  apply  new  knowledge.  This  may  encourage 
behavior  change.  Games  can  be  useful  feedback  tools  even  where  literacy  levels  are  high, 
since  they  provide  an  active,  interactive,  and  often  entertaining  means  of  obtaining 
information  about  what  was  learned  and  how  it  is  likely  to  be  applied  to  real-life  situations. 

In  using  games  to  evaluate  the  extent  to  which  new  knowledge,  attitudes,  or  skills 
have  been  learned,  it  is  very  important  that: 

•  The  games  are  played  in  small  groups  or  in  a  way  that  assures  that  most  or  all 
group  members  participate  —  so  the  learning  of  all  can  be  assessed. 
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•  A  sufficient  number  of  staff  or  other  observers  are  present  to  record  the 
individual  responses  of  the  participants,  so  that  as  little  information  as  possible 
is  missed.  Taping  can  be  helpful  if  the  system  is  of  sufficient  quality  that 
everyone  can  be  heard,  and  if  someone  systematically  extracts  the  appropriate 
information  from  the  tape.  (You  should  advise  participants  that  you  plan  to 
tape  their  discussions.) 

•  The  situation  —  game,  role  play,  or  case  study  —  is  carefully  planned  and 
explained  to  the  group.  It  is  important  that  the  facilitator  or  moderator  take 
time  at  the  beginning  of  the  session  to  thoroughly  explain  the  game  or  other 
activity  to  assure  that  all  participants  understand  what  to  do.  The  facilitator 
should  also  explain  any  "ground  rules"  that  all  participants  must  adhere  to 
(e.g.,  participants  must  be  respectful  of  one  another  and  refrain  from 
criticizing  each  other’s  comments,  opinions,  or  questions,  etc.).  If  the  group 
is  literate,  the  facilitator  can  write  down  the  main  points  and  "ground  rules"  of 
the  game  on  a  blackboard  or  easel  pad  for  easy  reference.  Before  beginning 
the  game,  the  facilitator  should  ask  if  there  are  any  questions,  and  even  ask  the 
group  questions  about  the  instructions  to  make  sure  they  understand. 

The  following  exercise  modifies  and  builds  upon  a  game  used  with  an  African- 
American  women’s  program  in  Chicago  run  by  the  Kupona  Network: 

►  Your  session  is  designed  to  teach  women  (or  some  other  target  group)  the  ways  in 
which  the  HIV  virus  can  be  transmitted.  At  the  beginning  of  the  session,  write  five 
names  on  the  board,  and  ask  members  of  the  group  to  work  in  pairs  to  identify  five 
different  ways  that  each  of  them  might  have  contracted  the  HIV  virus.  If  you  find 
that  no  one  is  identifying  perinatal  transmission,  you  might  point  to  one  name,  saying 
"This  person  has  AIDS,  but  has  never  had  sex,  never  used  injecting  drugs,  and  never 
had  a  transfusion;  how  did  s/he  get  AIDS?" 

A  non-participant  observer  should  be  recording  the  responses  of  each  pair.  Answers 
to  such  questions  will  provide  staff  a  good  sense  of  pre-session  knowledge  about  HIV 
transmission.  At  the  end  of  the  session,  divide  the  group  into  pairs  again  to  assess  whether 
they  have  learned  the  various  methods  of  transmission.  This  time,  staff  should  use  a  slightly 
more  complicated  version  of  the  game: 

►  Here  are  seven  people:  Miguel,  Juan,  Maria,  Jose,  Cecilia,  Jenny,  and  David.  Eight 
years  ago,  only  Miguel  was  HIV-positive;  now  all  seven  of  these  people  are  HIV¬ 
positive.  Explain  the  relationships  between  these  people  and  describe  how  each  of  the 
other  people  might  have  become  infected,  directly  or  indirectly,  from  Miguel. 

Include  each  different  type  of  transmission. 
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There  are  many  different  "answers"  to  this;  a  typical  explanation  would  be  as  follows: 

Miguel  had  sex  with  Juan  who  contracted  HIV  through  their  homosexual  relationship. 
Method:  anal  intercourse  between  two  men. 

Juan  was  a  person  who  had  both  male  and  female  sex  partners,  and  he  passed  the 
virus  on  to  his  girlfriend,  Maria,  through  their  sexual  relationship.  Method:  virus 
was  transmitted  from  a  man  to  a  woman  through  vaginal  intercourse. 

Before  the  blood  supply  was  being  routinely  tested,  Juan  gave  blood,  and  Cecilia  got 
the  HIV  virus  from  his  blood  when  she  had  an  emergency  transfusion  after  an  auto 
accident.  Method:  blood  transfusion. 

Maria  broke  up  with  Juan  and  married  Jose,  who  got  the  virus  from  her  through  a 
heterosexual  relationship.  Method:  virus  was  transmitted  from  a  woman  to  a  man 
through  vaginal  intercourse. 

Maria,  who  didn’t  know  she  was  HIV-positive,  got  pregnant  and  gave  birth  to  Jenny, 
who  was  born  with  the  HIV  virus.  Method:  perinatal  transmission  from  a  mother  to 
her  unborn  child. 

Jose  was  an  injecting  drug  user,  and  David  used  one  of  his  dirty  needles  and  got  the 
HIV  virus.  Method:  needle-sharing  by  injecting  drug  users. 

This  game  can  be  taken  one  step  further  if  you  have  been  teaching  about  positive  risk- 
reduction  behavior.  The  session  leader  might  discuss  one  set  of  responses  like  those  above, 
and  then  ask  the  group:  "How  could  each  of  these  transmissions  have  been  avoided?  — 
what  risk-reduction  actions  could  have  been  taken?"  Responses  should  include  the 
following: 


•  The  blood  supply  is  largely  safe  today.  However,  anyone  who  is  or  thinks 
s/he  might  be  HIV-positive  certainly  should  not  donate  blood. 

•  The  probability  of  transmission  through  sex  —  between  men  or  between  a  man 
and  a  woman  —  could  be  reduced  by  use  of  latex  condoms.  Reduction  of 
casual  sex  will  also  reduce  this  risk  factor. 

•  Using  bleach  to  sterilize  used  needles  and  using  new  needles  will  minimize  the 
probability  of  transmission  through  dirty  needles.  Getting  drug  treatment 
which  leads  to  elimination  of  injecting  drug  use  is,  of  course,  a  much  better 
long-term  solution. 


NCLR  AIDS  Center 
Understanding  Evaluation  Techniques 


43 


•  HIV  testing  would  tell  individuals  whether  or  not  they  are  HIV-positive,  and 
would  enable  HIV-positive  and  -negative  individuals  to  receive  counseling,  and 
minimize  transmission  possibilities  by  limiting  the  number  of  sexual  relation¬ 
ships  and  using  condoms. 

•  The  only  way  to  prevent  perinatal  transmission  is  for  an  HIV-positive  woman 
not  to  become  pregnant. 

5.  Case  Studies  and  Role  Plavs 

Another  active  feedback  method  is  the  use  of  case  studies  and  role  plays.  If 
a  session  is  designed  to  help  young  adults  and  teenagers  learn  about  HIV/STD  transmission 
and  about  ways  to  reduce  their  own  risk,  a  closing  role  play  might  put  the  participants  in 
pairs  or  other  small  groups  (four  or  five  persons  at  most)  and  give  them  a  case  study  and 
role  play.  For  example: 


You  are  a  young  woman  on  your  third  date  with  a  good-looking  guy.  He  has 
indicated  that  he  wants  to  begin  a  sexual  relationship.  You  know  that  he  was 
taking  drugs  a  couple  of  years  ago,  but  has  since  stopped.  What  do  you  do? 


You  might  have  the  entire  group  discuss  different  approaches  and  decide  what  they 
would  do.  Or  you  can  do  a  role  play  by  having  one  person  in  each  small  group  role  play  the 
woman,  another  the  man,  and  have  two  others  serve  as  observers,  responsible  for 
remembering  or  jotting  down  the  major  points  each  person  makes  in  the  role  play.  (You  can 
even  have  a  third  observer  focus  on  their  interaction,  non-verbal  communication,  etc.)  Staff 
should  be  observing  each  small  group  and  recording  the  extent  to  which  the  groups  apply 
positive  risk-reduction  techniques  which  reflect  or  build  on  material  presented  in  the  session. 

Another  approach  involves  presenting  statements  and  asking  small  groups  to  respond 
to  them.  The  statements  can  be  read  to  each  small  group  by  a  facilitator  if  literacy  rates  are 
low.  The  purpose  is  to  see  if  information  presented  in  the  session  was  learned  and  is  applied 
appropriately  to  real-life  situations.  For  example,  the  following  statements  might  be  used: 


Statement  from  a  man  whose  male  sex  partner  has  just  been  diagnosed  with  AIDS, 
and  who  says  he  doesn  ’t  need  to  get  an  HIV  test:  I  can’t  have  AIDS  because  I 
am  not  gay.  I  just  had  Dan  as  a  sex  partner  because  my  wife  lives  in  Mexico, 
and  I  didn’t  want  to  cheat  on  her.  We  have  three  children. 

Response  sought :  You  get  HIV/AIDS  because  of  your  behavior,  not  your 
values.  Any  man  who  has  sex  with  another  man  that  is  HIV-positive  is  at  risk 
for  HIV/AIDS  and  should  be  tested. 
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Statement  from  a  woman  whose  partner  is  an  injecting  drug  user:  I  am  afraid  he 
might  have  the  HIV  virus,  but  if  I  ask  him  to  wear  a  condom  he  will  get  very 
angry  at  me,  and  might  beat  me  or  leave  me. 

Response  sought:  Negotiating  safer  sex  practices  with  your  partner  can  be  a 
complicated  and  difficult  process.  Ultimately,  you  need  to  remember  that 
there  is  no  cure  for  HIY/AIDS,  so  preventing  this  disease  is  critical.  If  you 
have  unprotected  sex  with  someone  who  may  be  HIV-positive,  you  are  literally 
risking  your  life.  You  may  be  able  to  encourage  him  to  get  an  HIV/ AIDS 
test,  or  to  change  his  behavior,  to  stop  injecting  drugs,  or  to  use  only  clean 
needles.  You  may  be  able  to  persuade  him  to  use  a  condom  for  the  sake  of 
protecting  you  and  any  future  children,  or  argue  for  condom  use  as  an 
effective  birth  control  method.  Realistically,  your  choice  may  be  between 
breaking  off  the  relationship  and  risking  your  life. 


Television  dramas/stories  can  also  provide  an  effective  role  playing  situation  that  can 
help  your  target  group  learn  about  high-risk  behavior.  On  week  nights,  many  Hispanics, 
particularly  women,  like  to  watch  telenovelas  (soap  operas)  broadcast  on  the  Spanish- 
language  channels.  These  are  popular  forms  of  entertainment  and  serve  as  topics  of 
conversation.  Telenovelas  often  depict  real-life  situations  involving  problems  arising  in 
relationships  and  family.  HIV/STD  education  and  prevention  sessions  targeting  Hispanic 
women  could  develop  a  videotape  presentation  using  one  or  two  episodes  from  a  popular 
telenovela  for  discussion.  Staff  and  community  experts  should  first  preview  the  telenovela , 
identify  important  issues  around  HIV/STD  for  discussion,  and  prepare  structured  questions 
before  presenting  the  videotape  to  the  target  group.  The  responses  to  focused  questions  can 
provide  an  understanding  about  the  group’s  HIV/STD-related  knowledge,  including  myths 
and  stereotypes  about  AIDS.  The  videotape  could  be  presented  twice  during  a  session, 
before  and  after  the  group  has  received  HIV/STD  prevention  and  skills-building  information. 
Discussions  following  the  videotape  presentations  would  indicate  whether  the  information  has 
enhanced  the  group’s  knowledge  and  reduced  myths  and  stereotypes  about  HIV/STDs. 

An  exercise  following  the  video  could  have  participants  either  as  a  group  or  in  small 
groups  identify  characters  who  may  be  at  risk  of  HIV/STDs  and  explain  why  and  how  these 
characters  could  contract  the  disease.  Other  AIDS-related  issues  could  be  explored  as  well, 
such  as  the  importance  of  family  support  and  understanding  for  people  who  are  ill, 
information  about  community  health  and  mental  health  resources  for  persons  with 
HIV/ AIDS,  raising  confidence  and  self-esteem  in  relationships  —  all  critical  issues  that 
increase  awareness  and  can  reduce  the  incidence  of  HIV/ AIDS.  AIDS  educators  could  pose 
questions  based  on  characters  from  the  telenovelas  to  groups  for  discussion.  For  example: 
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Pepe  recently  discovered  that  he  had  tested  positive  for  HIV.  He  has  been  very 
depressed  and  has  not  revealed  his  HIV-positive  status  to  any  of  his  sex  partners 
or  family  members.  He  did  reveal  his  status  to  you  because  you  are  his  closest 
friend.  What  kind  of  advice  would  you  offer  to  urge  Pepe  to  notify  his  sex 
partners? 

Pepe  especially  fears  the  type  of  reaction  he  will  get  from  his  family  members. 
What  advice  could  you  provide  him?  Are  there  specific  places  you  would 
recommend  he  go  for  assistance? 


The  written  version  of  the  telenovela ,  th e  fotonovela,  is  also  a  common  form  of 
recreation  among  many  Hispanics,  particularly  women.  When  your  group  is  fluent  and 
literate  in  Spanish,  [he  fotonovela  can  be  used  as  an  educational  tool  in  seminars  on 
HIV/STDs.  Fotonovelas  are  short,  comic-book-sized  narratives  with  black  and  white  or 
cartoon-style  pictures.  The  narrative  tells  a  story  and  usually  focuses  on  two  or  three  main 
characters,  one  of  whom  may  be  at  risk  of  HIV  infection  or  may  have  contracted  AIDS. 
The  following  exercise  is  one  way  to  use  a  fotonovela  during  an  HIV/STD  education  and 
prevention  session: 


Allow  the  group  15-20  minutes  to  read  through  a  fotonovela  on  HIV/STDs.  Then 
have  the  group  come  up  with  at  least  three  main  characters  of  the  fotonovela. 
Break  up  the  group  into  three  small  groups:  one  group  will  be  Character  1;  the 
second  group,  Character  2;  and  the  third  group,  Character  3.  Each  group  should 
identify  specific  ways  in  which  the  group’s  particular  character  contributed  or 
helped  to: 

1.  Increase  or  decrease  either  their  own  or  someone  else’s  risk  for  HIV/STD 
transmission. 

2.  Dispel  misconceptions  or  myths  about  HIV/ AIDS. 

3.  Increase  awareness  of  other  STDs  besides  HIV/ AIDS. 
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The  following  are  some  examples: 


Carmelo  and  Juan  go  the  local  tavern  after  work.  After  several  hours  of  drinking, 
Juan  begins  to  flirt  with  a  stranger.  Juan  expresses  to  Carmelo  his  intent  to 
continue  having  fun  with  the  stranger  tonight.  Carmelo  gives  Juan  a  pat  on  the 
back  and  orders  another  round  of  drinks  for  the  three  of  them.  Carmelo’ s 
encouragement  to  continue  drinking  increases  the  likelihood  that  Juan  and  the 
stranger  will  have  unprotected  sex,  which  can  lead  to  infection  with  HIV/ AIDS  or 
other  STDs. 


During  a  conversation  between  Liona  and  Pablo,  Pablo  mentions  that  he  has 
stopped  swimming  in  pools  because  of  the  fear  that  he  will  get  AIDS.  Liona 
interrupts  and  tells  him  that  he  cannot  get  AIDS  by  swimming  in  pools,  it  is  the 
exchange  of  body  fluids  --  either  of  infected  semen  or  blood  —  that  leads  to 
infection  with  HIV/ AIDS.  In  this  case,  Liona  is  trying  to  dispel  misinformation 
about  the  transmission  of  HIV/AIDS. 


Linda  is  worried  about  her  good  friend  Melissa  who  has  had  unprotected  vaginal 
sex  with  three  different  partners  in  the  last  two  months.  After  Linda  raised  the 
subject  of  HIV/ AIDS,  Melissa  assured  Linda  that  none  of  her  sexual  partners  have 
used  injecting  drugs  or  have  had  sex  with  other  males  before.  Consequently, 
Melissa  feels  no  need  to  get  HIV-tested.  Linda  cautions  her  friend  that 
heterosexuals  can  transmit  AIDS,  and  also,  AIDS  is  one  of  many  diseases  that  are 
transmitted  sexually.  Other  non-HIV  sexually  transmitted  diseases  are  extremely 
common  and  if  left  untreated  can  cause  permanent  damage  to  reproductive  organs. 
Not  only  HIV,  but  also  diseases  such  as  gonorrhea  and  syphilis  could  have  been 
transmitted  to  her  sexual  partners  by  other  women,  thereby  placing  Melissa  at 
risk.  Linda  finally  succeeds  in  getting  her  friend  to  visit  an  STD  clinic.  In  this 
case,  Linda  is  trying  to  dispel  myths  about  the  transmission  of  HIV/AIDS  and 
increase  Melissa’s  awareness  of  other  STDs. 
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6.  Kev-Person  Interviews 


Key-person  or  key-informant  interviews  are  useful  for  obtaining  an  in-depth 
understanding  about  the  effectiveness  of  ongoing  interventions  or  key  issues  to  consider  in 
developing  or  improving  interventions.  This  method  involves  developing  an  interview  guide 
or  checklist  specifying  major  topics  and  issues  to  be  covered,  but  not  structured  questions. 
This  checklist  then  serves  as  the  guide  for  carrying  out  an  in-depth  interview  with  a  small 
number  of  individuals  who  have  had  personal  experiences  with,  and/or  are  knowledgeable 
about  the  problems  and  groups  being  targeted  by  a  program.  Key-person  interviews  can 
provide  a  quick  assessment  of:  (1)  the  issues  to  be  considered  in  designing  a  new  education 
and  prevention  approach,  (2)  how  a  prevention  approach  is  working,  or  (3)  major  variables 
to  investigate  further  in  an  informal  survey  or  pre-  and  post-test. 

Key-person  interviews  are  useful  only  when  the  individuals  interviewed  are  in  fact 
knowledgeable  and  willing  to  provide  detailed  information.  Although  the  interviews  cannot 
provide  information  for  generalization  to  a  larger  population,  they  should  provide  a  wide 
range  of  perspectives  on  the  program  and  the  issue  (e.g.,  key  issues  that  should  be 
considered  in  implementing  a  peer  AIDS  educator  program,  effective  HIV/STD  outreach 
methods  for  gay  Hispanic  youth,  etc.).  Therefore,  it  is  important  that  the  individuals  to  be 
interviewed  are  carefully  selected  to  assure  various  points  of  view,  experiences,  and 
concerns.  This  may  mean  people  of  varying  income  and  occupational  groups,  people  who 
are  considered  likely  to  engage  in,  or  be  knowledgeable  about,  different  types  of  high-risk 
behaviors,  and  staff  of  organizations  as  well  as  program  clients,  etc.  The  appropriate 
categories  will  depend  on  what  is  being  assessed. 

Because  the  interview  is  a  guided  discussion  without  closed-ended  questions,  it 
requires  an  interviewer  who  is  extremely  knowledgeable  about  the  project,  and  trained  to 
probe  for  information  without  presenting  his/her  views  in  a  way  which  might  bias  the 
responses.  The  interviewer  should  take  detailed  notes  of  the  responses  to  the  questions  and 
might  also  tape  the  session  for  back-up.  After  the  session,  the  notes  should  be  compiled  into 
a  final  interview  report.  It  may  be  necessary  to  call  the  individual  back  for  more 
information. 

A  series  of  key-person  interviews  will  generate  in-depth  qualitative  information  about 
a  project,  information  which  will  typically  need  to  be  supplemented  with  other  project  data 
such  as  focus  group  results,  post-session  assessments,  and  perhaps  informal  surveys.  The 
results  of  such  interviews  can  also  help  program  staff  select  the  issues  to  be  covered  in  focus 
groups,  assessments,  and  surveys,  and  can  help  frame  the  questions  appropriately.  The 
following  example  illustrates  the  benefits  of  this  approach: 
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CASE  STUDY 

Using  key  informants  to  obtain  feedback  on  a  new 
street  outreach  program 

Three  months  ago,  Project  REACH  implemented  a  three-year  HIV/STD  street 
outreach  program  targeting  Hispanic  injecting  drug  users  (IDUs)  in  the  Pilsen 
area.  As  part  of  their  evaluation  efforts,  staff  will  conduct  a  series  of  key-person 
interviews  throughout  the  year  to  get  feedback  on  their  approaches  and  strategies 
for  conducting  street  outreach.  The  first  set  of  interviews  will  assess  primarily  the 
information  being  disseminated  by  outreach  workers.  Staff  will  recruit  key 
informants  representing  former  prostitutes,  HIV  counselors,  recovering  drug 
addicts,  and  street  outreach  workers  from  other  agencies. 

Project  REACH  will  have  available  for  the  key  informants  the  written  versions  of 
the  types  of  information  generally  provided  to  all  clients  during  street  outreach 
sessions,  as  well  as  any  pamphlets  distributed.  In  addition,  key  informants  will 
view  a  videotape  showing  outreach  workers  role  playing  several  street  outreach 
sessions.  This  will  provide  key  informants  with  a  better  sense  of  Project 
REACH’S  street  outreach  sessions.  The  following  questions  will  be  used  as  an 
interview  guide: 

•  Are  the  outreach  workers  providing  the  kinds  of  information  that 
will  be  useful  to  Hispanic  IDUs? 

•  What  other  types  of  information  should  be  provided  to  Hispanic 
IDUs? 

•  What  do  you  think  about  the  information  being  delivered  by 
outreach  workers?  Is  it  understandable,  clear,  and  appropriate  for 
the  Hispanic  IDUs  living  in  the  Pilsen  area? 

•  Are  the  outreach  workers  providing  correct  information  about 
HIV/STDs,  ways  to  reduce  risk  of  HIV,  referrals,  risk  groups,  etc.? 

•  Are  outreach  workers  targeting  the  locations  where  IDUs  are  likely 
to  be  found? 
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F.  Community  Indicators 


It  is  also  important  that  an  HIV/STD  program  measure  community  exposure  and 
awareness  of  HIV/STD  prevention  messages.  While  it  is  very  difficult  to  measure  behavior 
change  related  to  HIV/STD  prevention  efforts  without  formal,  costly  KABB  studies,  some 
community  indicators  can  suggest  whether  behavior  change  is  likely,  or  even  if  it  is 
occurring.  The  following  are  examples: 

•  If  your  organization  puts  out  flyers  or  PSAs  telling  people  to  call  for 
information  about  HIV/STDs, 
have  the  staff  person  who 
answers  the  telephone  tally  the 
number  of  calls  received  daily 
and  weekly  for  the  duration  of 
the  campaign  and  ask  where 
people  got  the  telephone  number. 

This  will  provide  feedback  on  the 
strength  and  visibility  of  your 
message.  It  will  also  provide 
some  indication  as  to  the  extent 
of  community  need  for  HIV/STD 
information. 

•  If  a  community  outreach  effort  or 
media  campaign  urges  people  to 
get  tested,  HIV  testing  sites  can 
keep  track  of  the  number  of 
people  requesting  tests  before 
and  after  the  campaign. 

Information  about  how  people 
learned  about  the  testing  center 
should  also  be  sought  to  find  out  what  types  of  outreach  efforts  are  effective. 

•  Use  your  agency’s  AIDS/STD  hotline  to  develop  an  understanding  of  the  types 
of  information  needed  in  your  community.  Periodically,  conduct  an  analysis 
of  why  people  call  your  AIDS/STD  hotline  (e.g.,  to  find  out  symptoms  of  HIV 
and  other  STDs,  obtain  HIV/STD  testing  information,  ask  HIV/STD 
transmission  questions,  etc.).  Document  and  use  this  background  information 
when  implementing  new  HIV/STD  projects  and  when  developing/revising 
HIV/STD-related  materials. 


THREE  WAYS  TO  ASSESS 

COMMUNITY  AWARENESS  OF 
HIV/STDs 

1 .  Tally  the  number  and  type  of 
information  requests  made  to 
your  AIDS/STD  hotline  or 
agency  on  a  monthly  basis. 

2.  Track  the  number  of 
HIV/STD  tests  administered 
monthly  at  a  local  testing  site. 

3.  Collaborate  with  local  drug 
stores  and  supermarkets  to 
track  condom  sales  on  a 
quarterly  basis. 
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•  Some  organizations  have  strong  visibility  in  the  community,  particularly  small 
businesses  such  as  local  drug  stores,  supermarkets,  and  restaurants  operating  in 
a  particular  neighborhood.  Collaborate  with  community  business  owners  and 
orient  supermarket  and  drug  store  owners  to  the  importance  and  need  for 
community  HIV/STD  education  efforts.  Then  periodically  ask  them  to  keep 
track  of  condom  sales,  emphasizing  your  organization’s  interest  in  preventing 
the  spread  of  AIDS  and  other  STDs  in  the  community  by  getting  more  people 
to  adopt  condom  use. 

Community  indicators  can  also  provide  a  longer-term  indication  of  the  overall  impact 
of  HIV/STD  education  and  prevention  efforts.  Suppose  your  organization  is  about  to  begin  a 
major  project  in  a  defined  community  or  neighborhood.  Have  baseline  data  available  before 
the  project  starts  concerning  number  of  calls  requesting  HIV/STD  information,  number  of 
people  requesting  HIV  tests,  number  of  requests  for  flyers/brochures,  etc.,  per  week  or 
month.  Continue  to  collect  and  report  the  data  after  the  project  begins.  Graphing  changes  in 
requests  for  information,  products,  and  services  over  time  suggests  whether  prevention 
efforts  are  having  an  effect.  Also,  have  community  health  centers  or  other  social  service 
agencies  you  collaborate  with  ask  a  sample  of  their  callers/clients  about  how  they  learned 
about  the  service.  This  will  give  you  more  information  about  the  effects  of  your  project’s 
specific  interventions. 

Aggregating  these  types  of  data  will  provide  a  valuable  indication,  but  not  a  clear 
measure,  of  the  effects  of  your  prevention  efforts.  Because  these  are  community  measures, 
and  people  are  exposed  to  many  different  messages  from  a  variety  of  sources,  it  is  not 
possible  to  determine  a  cause-and-effect  relationship  between  project  interventions  and 
observed  changes  in  community  awareness  or  actions. 

G.  Administrative  Recordkeeping 

Since  the  basis  for  all  evaluation  is  information,  a  consistently  used  recordkeeping 
system  is  an  essential  component  of  any  evaluation  system.  Basic  recordkeeping  forms 
should  be  maintained  for  all  HIV/STD  projects.  Most  ongoing  publicly  funded  HIV/STD 
prevention  programs  have  a  required  set  of  forms  which  must  be  completed  regularly, 
tabulated  monthly  or  quarterly,  and  sent  to  the  funder  in  aggregated  form.  When  there  is 
more  than  one  funding  source  for  a  project,  each  source  may  require  the  completion  of 
different  forms.  At  present,  no  standardized  reporting  system  exists  for  HIV/STD  education 
and  prevention  programs,  and  it  may  be  difficult  to  develop  a  single  set  of  forms  for  the 
variety  of  ongoing  interventions. 

Developing  new  or  revising  existing  forms  for  your  programs  and  activities  requires 
considerable  planning  and  some  information  gathering.  The  forms  that  you  develop  or 
modify  will  most  likely  be  around  for  some  time  so  it  is  important  that  the  time  invested  in 
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this  process  is  spent  wisely.  Remember  that  the  information  generated  by  forms  is  used  to 
produce  many  layers  of  information.  If  your  forms  are  poorly  designed  initially,  you  will 
need  to  repeat  the  same  efforts  in  the  future  when  you  could  be  proceeding  into  the  next 
phase  of  the  project.  The  box  below  contains  some  questions  you  should  consider  as  you 
design  forms: 


QUESTIONS  TO  CONSIDER  WHEN  DESIGNING  FORMS 

•  What  types  of  program  information  are  mandatory  —  required  by  funders, 
state  or  local  health  departments,  federal  agencies? 

•  Who  or  what  agencies  will  be  receiving  the  information? 

•  What  other  program-specific  data  need  to  be  collected  for  internal  use,  and 
why  are  these  data  important? 

•  How  often  must  you  collect  the  information  (e.g.,  weekly,  monthly, 
quarterly,  annually)? 

•  Can  one  standard  form  be  developed  for  sources  requesting  similar 
information? 

•  Who  will  complete  the  forms?  Will  it  be  clients,  or  intake  or  other  staff? 

•  Who  will  tabulate  the  information  or  log  it  into  a  data  base? 

•  Who  will  design  the  form?  Will  you  pilot  test  your  form? 


Since  it  is  important  to  track  the  different  types  of  interventions  conducted  by  an 
organization,  it  may  be  necessary  to  have  separate  forms  for  each  type  of  intervention.  In 
any  health-related  education  and  prevention  program,  two  types  of  information  must  be 
recorded: 

•  Client/participant  data.  These  include  the  number  and  characteristics  of 

participants  in  various  interventions  —  e.g.,  number  of  youth,  women,  IDUs, 
health  providers,  community  leaders,  or  other  target  group  members  reached 
through  the  project,  and  their  most  relevant  characteristics,  including  gender, 
age,  and  ethnicity/nationality. 
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•  Program/intervention-based  data.  These  include  the  number  and  types  of 
interventions  carried  out  by  the  project,  such  as  street  outreach  activities, 
formal  education  sessions,  informal  presentations,  PSAs  prepared  and 
disseminated  and  number  of  times  used  or  audience  size  reached,  materials 
developed  and  number  of  copies  disseminated.  These  records  should  also 
show  which  staff  carried  out  each  intervention  and  the  date  of  each 
intervention. 

Client/participant  information  is  usually  collected  by  having  clients  or  participants 
complete  some  type  of  self-identification  form  such  as  a  client  registration/intake  form. 

These  forms  are  filled  out  one  time  only,  and  are  used  for  maintaining  basic  information 
about  clients/participants  on  file.  Depending  on  how  much  information  you  need,  forms  can 
be  short  and  brief  or  comprehensive,  asking  for  name,  address,  demographic  information, 
place  of  employment,  occupation,  referral  source,  health  status  information,  and  in  some 
cases  billing  information.  When  the  project  is  multi-service  and  serves  many  clients,  it  is 
important  to  continue  collecting  information  after  initial  client  contact  to  document  the  types 
of  services  the  client  receives.  Some  organizations  use  client  encounter  forms  for 
documenting  such  information.  Client  encounter  forms  should  be  filled  out  each  time  the 
client  receives  a  service,  to  facilitate  monthly  or  quarterly  tallying  of  services.  These  forms 
should  ask  about  the  type  of  service  provided,  where  the  service  was  provided,  and  how  it 
was  provided  (See  Worksheet  D). 

Forms  for  gathering  program-related  information  should  be  tailored  to  the  type  of 
activity  being  conducted.  For  example,  forms  for  street  outreach  sessions  will  need  to  be 
different  from  HIV/STD  presentation  forms.  For  presentations,  it  is  important  to  maintain 
information  about  the  site,  type,  and  length  of  the  presentation,  as  well  as  any  restrictions  the 
site  contact  person  might  require  presenters  to  adhere  to  (See  Worksheet  E).  Street  outreach 
forms,  however,  should  be  brief,  since  outreach  workers  will  have  to  complete  these  in 
public  places.  Only  the  most  pertinent  information  should  be  sought,  e.g.,  information  about 
high-risk  behaviors,  whether  clients  have  been  tested  for  HIV/STDs,  and  the  types  of 
referrals  made  (See  Worksheet  F). 

Client  and  program  information  forms  are  the  basis  of  your  program’s  monthly, 
quarterly,  and  annual  reports.  The  more  clear  and  comprehensive  the  forms,  the  easier  it 
will  be  to  analyze  and  aggregate  the  information  needed  for  periodic  reports.  Except  for  a 
few  narrative  analyses  of  the  data,  the  monthly  or  quarterly  report  forms  should  be  primarily 
fill-in-the-blank  types.  Staff  should  only  have  to  Fill  in  numbers  and  dates.  All  staff  should 
be  trained  to  use  the  forms,  and  their  completion  and  submission  should  be  an  absolute 
requirement. 
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TO  DEVELOP  USEFUL  RECORDKEEPING  FORMS 

•  Identify  the  major  activities  and  interventions  included  in  the  project 
design. 

•  Review  process  objectives  (e.g.,  number  of  each  type  of  intervention  to  be 
carried  out  and  number  and  characteristics  of  people  to  be  reached),  and  be 
sure  forms  ask  for  this  information. 

•  When  interventions  are  very  different,  develop  separate  forms  for  each  type 
of  intervention  or  activity. 

•  Be  sure  all  forms  are  primarily  the  fill-in-the-blank  types. 

•  Have  all  staff  review  draft  forms  before  finalizing  them. 

•  Have  staff  pilot  test  the  use  of  the  forms,  then  review  what  worked  and 
what  didn’t  and  make  needed  revisions. 

•  Tabulate  the  information  monthly,  and  compare  results  with  the  kinds  of 
information  needed  to  assess  progress  towards  process  objectives  and  goals. 
Further  revisions  may  be  necessary  based  on  this  review. 

•  Whenever  you  add  or  change  an  intervention,  be  sure  to  revise  the  forms 
as  needed. 


H.  Narrative  Case  Studies 

Community-based  organizations  must  develop  the  habit  of  maintaining  written 
narrative  records  of  program  implementation,  progress,  and  problems.  Since  most  HIV/STD 
education  and  prevention  projects  have  been  funded  as  demonstration  efforts,  documentation 
of  the  project  is  essential  for  two  reasons:  (1)  to  enable  funders  to  understand  the  full 
process  involved  in  launching  the  project,  including  unexpected  problems  and  obstacles,  and 
(2)  to  enable  replication  of  the  project  in  other  locations. 

Narrative  case  studies  are  the  equivalent  of  writing  project  diaries.  When  done 
systematically,  case  studies  should  provide  descriptive  information  about  a  project,  beginning 
with  the  planning  and  covering  implementation  and  ongoing  operations. 
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Where  possible,  it  is  worth  budgeting  and  allocating  some  funds  to  conduct  an 
effective  case  study.  One  staff  person  should  be  given  primary  responsibility  for  this  task. 
Ideally,  this  person  should  be  relatively  detached  from  the  actual  intervention  aspects  of  the 
project,  and  should  not  be  a  staff  member  whose  primary  goal  is  to  make  sure  the  project 
succeeds.  If  funding  for  a  special  "recorder"  is  not  available,  a  key  staff  member  can  take 
on  this  duty.  The  person  should  be  at  liberty  to  document  the  negative  as  well  as  the 
positive  experiences  of  the  project.  If  an  outsider,  the  "case  study  keeper"  will  need  to  meet 
frequently  with  key  staff  on  an  individual  and  group  basis  to  be  kept  informed  about  the 
activities  that  are  taking  place.  This  person  should  also  prepare  a  list  of  specific,  open-ended 
questions  in  advance  of  the  meetings.  These  questions  might  include: 

•  What  major  tasks  were  involved  in  implementing  this  project? 

•  What  types  of  staff  were  needed  to  carry  out  activities  of  the  project? 

•  What  types  of  interventions  were  used  (e.g.,  street  outreach,  in-house 

presentations,  threatrical  presentations,  peer  education,  etc.)? 

•  Who  were  the  target  groups  of  the  interventions? 

•  What  methods  were  used  to  reach  desired  target  groups?  Were  some  methods 

more  effective  than  others?  Which  one(s)  and  why? 

•  What  kinds  of  problems/obstacles  were  encountered  in  doing  outreach  to  target 
groups? 

•  If  more  than  one  intervention  was  used,  which  showed  the  most  benefit  and 
why? 

•  What  major  problems/obstacles  were  encountered  in  carrying  out 
interventions? 

•  What  were  the  reactions  of  the  target  groups  to  the  interventions?  What 
aspects  did  they  seem  to  enjoy  the  most/least? 

•  What  major  programmatic  problems  were  encountered? 

•  How  were  these  problems  resolved  or  minimized?  Describe  the  resolution 
process:  Who  was  involved  from  within  the  project?  Outside  of  the  project? 

•  Did  external  factors  (factors  outside  the  organization)  contribute  to  the  problem 
(e.g.,  unexpected  budget  cuts,  emergencies,  etc.)? 
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•  What  major  hurdles/challenges  were  overcome? 

•  What  advice/recommendations  would  you  give  to  another  community-based 
organization  about  to  launch  a  project  similar  to  the  one  at  your  agency? 

I.  Planned  Variation 

Uncertainty  often  exists  about  the  efficacy  of  various  intervention  approaches  in  new 
or  ongoing  projects.  The  only  way  to  learn  which  of  two  interventions  is  most  appropriate  is 
to  test  two  or  more  approaches.  Community-based  organizations  can  run  side-by-side 
interventions  for  a  period  of  time  to  determine  which  intervention  shows  the  most  benefit 
with  a  particular  target  group.  This  strategy,  referred  to  as  planned  variation,  involves 
random  assignment  of  one  group  of  clients  having  similar  characteristics  (e.g.,  similar  age, 
racial/ethnic  background,  educational  level,  etc.)  to  one  type  of  intervention  and  another 
group,  having  similar  characteristics  as  the  former  group,  to  a  different  intervention  for  a 
specific  period  of  time.  Since  it  can  be  difficult  to  carry  out  two  separate  interventions  with 
two  groups  at  the  same  time,  you  can  also  test  the  new  approach  by  using  it  in  place  of  the 
standard  approach  for  a  short  period  of  time  (e.g.,  three  weeks)  and  then  continue  using  the 
standard  approach.  Planned  variation  can  be  conducted  several  ways: 

Alternative  approaches:  Two  different  approaches  for  conducting  HIV/STD 
prevention  sessions  can  be  tried  with  similar  target  groups.  For  example,  Approach  A  and 
Approach  B  can  be  tried  on  two  groups  of  women,  all  approximately  the  same  age  and  with 
similar  racial/ethnic  backgrounds  to  see  which  approach,  A  or  B,  results  in  increased 
learning.  One  way  to  assess  which  educational  approach  works  better  is  to  administer  pre- 
and  post-tests  to  participants  in  both  groups.  These  tests  should  be  administered  to  each 
group  separately,  and  results  analyzed  independent  of  one  another  to  find  out  what  progress 
participants  of  each  group  made.  Answers  to  post-tests  from  one  group  should  then  be 
compared  with  those  of  the  other  group  to  identify  which  approach  showed  the  most  benefit. 
Staff  should  carefully  review  and  discuss  the  results,  and  should  try  the  favorable  approach 
once  more  before  making  any  program  changes.  The  results  of  these  trials  indicate  the 
approach  that  may  work  best  on  this  group  of  women  and  are  not  conclusive  for  the  general 
population. 

Augmenting  services:  A  standard  approach  can  be  tested  alongside  an  augmented 
version  to  see  whether  the  augmented  approach  seems  to  increase  learning  or  other  positive 
measures.  For  example,  augmenting  a  standard  counseling  session  could  involve  spending  an 
additional  15  minutes  on  a  particular  topic.  Or,  when  an  AIDS  educator  discusses  risk 
behaviors  for  AIDS,  the  educator  can  spend  some  extra  time  discussing  the  most  common 
STDs,  and  their  role  in  facilitating  HIV  infection.  At  the  end  of  a  trial  period,  two  focus 
groups  could  be  organized  for  an  in-depth  discussion  about  each  approach.  One  focus  group 
would  consist  of  several  participants  exposed  to  one  approach,  the  other  focus  group  would 
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consist  of  participants  exposed  to  the  other  approach.  Once  focus  groups  have  been 
conducted  and  notes  written  up,  project  staff  should  meet  to  discuss  the  results  of  each  focus 
group  and  tentatively  conclude  which  approach  worked  best. 

Adding  new  elements:  Add  new  elements  to  an  intervention  and  compare  against  the 
standard  intervention  using  two  groups.  Having  two  separate  comparison  groups  can  ensure 
that  at  the  end  of  the  trial  period,  participants  can  provide  focused  feedback  on  the  strengths 
and  weaknesses  of  that  one  intervention.  These  two  comparison  groups  can  be  run 
concurrently  or  at  different  times  depending  on  resource  and  staff  availability.  For  example, 
a  new  element  to  a  home-based  AIDS  education  session  for  women  could  include  a  20- 
minute  rap  session  to  help  women  get  acquainted  and  feel  comfortable  before  the  session 
begins;  the  standard  approach  would  have  the  facilitator  do  a  brief  introduction  of  the 
purpose  for  the  session,  have  everyone  else  do  introductions,  and  proceed  directly  into  the 
session;  alternatively,  a  new  element  for  the  session  might  include  a  role  play  or  a  game.  In 
order  for  planned  variation  to  be  effective,  staff  must  ensure  that  participants  remain  in  the 
assigned  intervention  for  the  duration  of  the  planned  variation  trial.  After  using  the  new 
approach  for  a  specified  period  of  time,  continue  using  the  standard  approach,  keeping  good 
documentation  of  the  sessions  during  the  specified  trial  period.  At  the  end  of  each  session, 
obtain  feedback  from  the  women  using  a  post-session  assessment  form,  informal  survey,  or 
oral  interview.  After  the  home-based  sessions  have  been  tested,  compare  feedback  from  that 
session  with  feedback  from  the  standard  sessions  to  see  which  one  women  liked  best  and 
learned  from  the  most. 
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V.  SELECTING  THE  APPROPRIATE  EVALUATION 
METHODS  FOR  HIV/STD  INTERVENTIONS 

The  purpose  of  this  section  is  to  demonstrate  how  the  various  evaluation  techniques 
discussed  earlier  can  be  combined  and  applied  to  certain  HIV/STD  education  and  prevention 
activities.  While  the  approaches  for  carrying  out  HIV/STD  activities  vary  across  agencies, 
all  programs  have  similar  basic  "ingredients"  which  should  be  assessed  regularly  in  order  for 
programs  to  be  successful. 

When  developing  an  evaluation  plan  for  your  project,  plan  to  use  not  just  one,  but 
several  evaluation  techniques.  Combining  various  techniques  or  "building  blocks"  to  obtain 
information  about  a  project  can  provide  a  more  in-depth  understanding  of  how  a  program  or 
an  activity  is  working.  For  example,  using  focus  groups  and  informal  surveys  to  assess  how 
well  a  high-school-based  AIDS  education  program  is  going  can  yield  more  information  about 
the  activity  than  only  using  informal  surveys.  The  information  generated  from  using  more 
than  one  assessment  technique  will  also  improve  the  reliability  of  the  evaluation. 

Three  types  of  HIV/STD  prevention  and  education  activities  are  used  as  examples 
below:  media  and  community  outreach  campaigns,  informal  or  street  outreach  interventions, 
and  HIV  counseling  and  testing  programs.  Under  each  type  of  activity,  aspects  of  the 
activity  which  should  be  evaluated  are  identified,  followed  by  suggested  evaluation 
techniques. 

A.  Media  and  Community  HIV/AJDS  Outreach  Campaigns 

Media  and  community  outreach  campaigns  are  often  used  to  reach  large  audiences 
with  education  and  prevention  messages  about  health  and  HIV/STD-related  issues.  The 
messages  are  often  designed  to  reach  specific  population  groups  such  as  teenagers, 
farmworkers,  drug  users,  limited-English-proficient  adults,  or  women.  Media  campaigns  can 
include  public  service  announcements  (PSAs)  on  television  or  radio,  newspaper 
advertisements,  press  conferences  (often  used  to  announce  a  campaign),  and  special  programs 
such  as  talk  shows.  Community  outreach  campaigns,  which  may  occur  alone  or  in 
coordination  with  media  campaigns,  often  use  buttons,  tee-shirts,  flyers,  posters,  and 
resource  kits  to  convey  messages  to  a  target  group.  These  campaigns  tend  to  reach  smaller 
numbers  of  people  than  media  campaigns. 

Key  Areas  for  Evaluation: 

Some  important  areas  of  a  media  and/or  community  outreach  campaign  which  should 
be  evaluated  include: 
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•  What  is  the  message,  and  how  will  it  be  delivered?  Make  sure  the  concept 
or  message  to  be  transmitted  is  correct  and  to  the  point.  Explore  the  options 
for  conveying  the  message  and  select  the  most  appropriate  media  or  outreach 
approach  —  e.g.,  a  message  encouraging  IDUs  to  come  for  HIV/STD  testing 
at  an  anonymous  testing  site,  conveyed  through  Spanish-language  radio  PSAs. 

•  Materials  for  distribution.  All  visual  or  written  intervention  materials  — 
such  as  press  releases,  videos,  buttons,  posters,  resource  kits,  bumper  stickers, 
and  flyers  should  undergo  review  for  consistency  of  message,  technical 
accuracy,  clarity,  and  cultural  appropriateness. 

•  Whether  the  desired  target  group  is  being  reached.  Assessing  the  extent  to 
which  the  campaign  actually  reaches  the  intended  number  of  targeted 
individuals  is  important;  for  example,  if  a  television  PSA  on  STDs  is  supposed 
to  reach  at  least  100,000  Hispanic  youth,  verify  that  this  really  happened  by 
checking  Nielsen*  and/or  STAR**  reports  which  indicate  the  number  of 
households  reached  and  the  audience  size  for  the  PSA. 

•  Effects  of  the  campaign.  Both  positive  and  negative  effects  of  the  campaign 
on  the  target  groups  should  be  documented  and  assessed.  For  example,  if  you 
want  to  assess  whether  people  exposed  to  a  radio  PSA  urging  HIV-testing  for 
individuals  practicing  high-risk  behaviors  actually  got  tested,  contact  the 
testing  site(s)  referred  to  in  the  PSA  and  have  them  ask  persons  how  they 
found  out  about  the  testing  site(s). 

Evaluating  Project  Design: 

If  you  are  in  the  planning  stages  of  a  media  or  community  outreach  campaign,  the 
following  evaluation  techniques  can  be  useful  for  obtaining  feedback: 

•  Expert  reviews.  Media,  graphics,  public  relations,  health,  AIDS,  and  STD 
experts;  staff  of  local  and  national  Hispanic  organizations;  and  community 
volunteers  can  be  sought  to  review  the  proposed  content  of  messages,  layout, 


*  Nielsen  Media  Research  conducts  national/local  TV  and  cable  research  services.  For  more 
information,  write  to:  Nielsen  Media  Research,  1290  Avenue  of  the  Americas,  New  York, 
New  York,  10104,  or  call  (212)  708-7500. 

**  Strategy  Research  Corporation  provides  national/local  radio  and  TV  Hispanic  audience  rating 
services,  conducts  Hispanic  market  studies,  station  profile  and  other  studies.  For  more 
information,  write  to:  Strategy  Research  Corp.,  100  NW  37th  Avenue,  Miami,  Florida, 
33125,  or  call  (305)  649-5400. 
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format,  accuracy,  language  and  cultural  appropriateness;  experts  can  also 
suggest  other  names  and  organizations  to  contact  for  additional  assistance. 

•  Focus  groups.  Well-organized  focus  groups  can  reveal  reactions  of  target 
group  members  to  proposed  messages,  provide  a  better  understanding  of  the 
information  needs  of  the  target  group,  and  suggest  feedback  on  specific 
language,  approaches,  and  materials. 

•  Community  interviews.  These  interviews  can  provide  general  feedback  on 
the  proposed  content  and  appropriateness  of  written/visual  materials  and 
strategies  for  the  campaign.  Staff  can  use  "captive"  audiences  to  collect  this 
information  from  groups  of  high  school  students,  clients  of  drug  and  alcohol 
treatment  centers,  and  clients  of  community-based  organizations. 

Evaluating  Project  Implementation  and  Outcomes: 

When  a  media  or  community  outreach  project  has  been  ongoing  for  some  time  and 
you  want  to  assess  its  implementation  and  outcome,  you  can  use  some  of  the  following 
evaluation  techniques: 

•  Documentation.  Track  all  the  materials  disseminated  during  the  campaign, 
including  quantity  of  flyers,  posters,  PSA  tapes,  and  other  materials;  maintain 
an  updated  list  of  all  the  places  materials  have  been  sent;  record  the  names  of 
radio  and  TV  stations  which  have  received  PSA  tapes  and  track  which  stations 
have  agreed  to  broadcast  the  tapes. 

•  Broadcast  and  print  media  reporting  of  use  of  PSA.  Ask  station  managers 
to  record  and  report  the  number  of  uses  of  a  PSA.  Arbitron  reports*  will 
indicate  the  number  of  households  who  were  tuned  in  to  the  radio  at  specific 
times,  and  Nielsen  data  will  do  the  same  for  television.  These  reports  also 
indicate  the  size  of  the  audience  for  the  PSA.  For  example,  they  can  indicate 
that  100,000  homes  were  tuned  in  and  saw  a  PSA  on  television.  You  can  also 
ask  magazines  and  newspapers  carrying  your  advertisements  for  the  date  of  the 
issue,  the  page  number  of  your  ad,  as  well  as  the  circulation  of  the  newspaper 
or  magazine;  many  magazines  also  have  subscriber  profiles. 

•  Community  measures.  If  the  campaign  asks  people  to  call  an  AIDS  or  STD 
hotline  for  more  information,  arrange  to  tally  the  number  of  telephone  calls 


The  Arbitron  Company  provides  television  and  cable  ratings,  commercial  monitoring  reports, 
and  product  sales  tracking  information.  For  more  information,  write  to:  The  Arbitron 
Company,  142  W.  57th  Street,  New  York,  New  York,  10019,  or  call  (212)  887-1300. 
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•  Community  measures.  If  the  campaign  asks  people  to  call  an  AIDS  or  STD 
hotline  for  more  information,  arrange  to  tally  the  number  of  telephone  calls 
made  to  the  hotline  several  weeks  before,  during,  and  after  the  campaign; 
operators  should  ask  callers  where  they  heard  about  the  hotline.  Similar 
documentation  can  be  done  by  HIV/STD  testing  sites  where  a  campaign  urges 
people  to  get  tested.  In  places  where  certain  stores  are  a  major  source  of 
condoms,  condom  sales  at  local  pharmacies  can  be  tracked.  Analyzing  such 
information  will  give  some  indication  about  whether  the  campaign  had  any 
effects  on  the  community. 

•  Key-person  interviews.  Inform  local  agencies  about  your  HIV/STD  campaign 
efforts.  Ask  agencies  who  work  directly  with  your  campaign’s  target  groups 
to  ask  their  clients  whether  they  know  about  the  campaign,  how  they  heard 
about  it,  their  opinions,  and  what  they  liked  and  disliked  about  the  campaign. 
Then  schedule  a  time  to  interview  the  agency  staff. 

•  Informal  surveys.  At  the  end  of  a  campaign,  visit  local  agencies  (with  prior 
consent  of  local  agency  staff)  where  you  are  likely  to  find  individuals 
representing  the  target  group  of  your  campaign  and  interview  clients 
informally,  either  orally  or  using  short  surveys.  You  can  survey  a  specified 
number  of  individuals  at  each  of  these  settings  —  perhaps  every  third  person 
who  enters  for  services  on  a  given  day.  Ask  clients  if  they  heard  about  or  saw 
the  campaign,  and  if  so,  what  new  information  was  learned  and  what,  if  any, 
actions  were  taken  as  a  result  of  hearing  campaign  messages. 

B.  Informal  or  Street  Outreach  Interventions 

Street  outreach  is  perhaps  the  most  challenging  of  all  HIV/AIDS  education  and 
prevention  activities  to  evaluate.  The  targeted  groups  are  hard-to-reach;  they  tend  not  to  go 
to  "typical"  settings  for  help  such  as  hospitals,  clinics,  and  drug  treatment  centers,  making 
traditional  methods  of  education  ineffective.  Yet,  these  groups  are  most  in  need  of  HIV/STD 
education  and  prevention  efforts  because  of  high-risk  behaviors  such  as  injecting  and  other 
drug  use  and  prostitution.  There  is  little  control  over  the  settings  in  which  street  outreach 
takes  place,  and  conducting  follow-up  can  at  best  be  "hit  or  miss"  —  contacts  may  not  stay  in 
the  settings  where  they  were  first  found.  Street  outreach  is  usually  an  informal,  one-time 
contact  activity  that  relies  more  on  one-to-one  personal  interaction  than  on  written  materials. 
The  outreach  approach,  length  of  time,  and  content  of  messages  may  vary  widely  from  one 
outreach  worker  to  another.  Regardless  of  these  variations,  almost  every  street  outreach 
intervention  has  certain  common  features  which  can  and  should  be  evaluated. 
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Key  Areas  for  Evaluation: 


Some  important  areas  of  informal  or  street  outreach  interventions  which  should  be 
evaluated  include: 

•  Whether  the  desired  target  group  is  being  reached.  Street  outreach  should 
be  targeting  individuals  whose  behavior  places  them  at  high  risk  of  HIV/STD 
infection.  Make  sure  to  assess  how  well  your  outreach  strategies  are  reaching 
these  groups. 

•  Content  of  information.  The  time  spent  with  each  street  contact  is  often 
insufficient  for  outreach  workers  to  provide  all  the  important  information  about 
HIV/STDs.  Outreach  workers  must  therefore  communicate  accurately  the  key 
points  about  HIV/STDs,  such  as  the  specific  behaviors  which  can  lead  to 
HIV/STD  infection,  methods  for  reducing  HIV/STD  risk,  HIV  and  STD 
diagnostic  and  testing  information,  and  referrals  for  alcohol  and/or  drug 
treatment. 

•  Delivery  of  messages.  It’s  important  to  assess  whether  outreach  workers  are 
delivering  HIV/STD  messages  in  a  clear,  comprehensive,  sensitive,  culturally 
appropriate,  and  comfortable  manner. 

•  Effects  of  messages  on  individual  behaviors.  While  this  is  by  far  the  most 
difficult  aspect  of  street  outreach  to  evaluate,  it  is  very  important  to  try  to  find 
out  how  individuals  who  were  reached  responded  —  whether  they  are  getting 
tested,  seeking  counseling,  enrolling  in  alcohol  and/or  drug  treatment 
programs,  and/or  asking  for  more  information  about  HIV/STDs. 

Evaluating  Project  Design: 

If  you  are  planning  an  informal  or  street  outreach  intervention,  use  the  following 
techniques  to  obtain  feedback  on  your  program  plans,  and  to  enable  you  to  make  any  changes 
before  its  full  implementation: 

•  Informal  surveys.  Surveying  individuals  from  the  community  who  are 
representative  of  your  target  group  can  offer  valuable  advice  about  the  best 
locations  for  reaching  people  like  themselves;  they  can  suggest  the  major 
points  that  need  to  be  communicated  as  well  as  effective  methods  or  styles  for 
establishing  rapport  with  individuals.  Usually,  you  will  want  to  use  a  brief 
survey  form  or  even  a  tape  recorder  for  these  interviews.  Tabulate  responses 
to  close-ended  questions,  and  list  hints  and  special  concerns. 
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•  Key-person  interviews.  Consider  interviewing  heads  of  local  social  service 
agencies,  drug  abuse  counselors,  community  volunteers,  and  neighborhood 
activists  for  information  and  insight  on  cultural  issues  to  be  considered  in 
doing  outreach,  unique  problems  and  needs  of  the  target  groups,  tips  on  how 
best  to  approach  certain  target  groups,  and  where  to  find  various  target 
groups.  A  semi-structured  interview  guide  and  a  tape  recorder  are  useful  for 
such  interviews. 

•  Documentation.  The  design  of  forms  for  documenting  street  outreach 
contacts  is  important.  Forms  should  be  brief  to  enable  workers  to  complete 
them  quickly.  Yet,  they  should  generate  important  information  about  street 
contacts.  Forms  should  ask  for  the  number  of  street  contacts  made,  types  of 
high-risk  behaviors,  characteristics  such  as  gender  and  approximate  age,  and 
location.  Internal  information  systems  should  also  be  in  place  before  the 
program  begins.  Assign  each  staff  person  responsibility  for  completing  certain 
tasks  such  as  collecting  street  outreach  forms,  documenting  aggregate  data  on  a 
monthly  or  quarterly  basis,  producing  summary  reports,  and  disseminating  the 
information.  Equally  important,  carry  out  early  reviews  and  obtain  street 
worker  feedback  to  be  sure  that  forms  are  being  used. 

Evaluating  Project  Implementation  and  Outcome: 

When  an  informal  or  street  outreach  intervention  has  been  ongoing  for  some  time, 
assess  its  implementation  and  outcome  using  some  of  the  following  techniques: 

•  Community  indicators.  If  you  refer  people  for  counseling,  testing,  or 
HIV/STD  information,  see  if  they  followed  your  advice.  Assess  the  extent  to 
which  the  sites  are  experiencing  a  rise  in  clients.  Sites  may  include 
community  social  service  agencies,  pharmacies,  health  centers,  STD  clinics, 
and  HIV  testing  sites.  If  you  have  referred  individuals  to  an  AIDS  hotline 
telephone  number,  find  out  if  increases  in  the  number  of  calls  have  occurred 
since  your  project  began.  Always  try  to  get  referral  sites  to  ask  new  clients 
and  contacts  who  told  them  to  come,  or  how  they  found  out  about  the  site. 

•  Documentation.  Maintain  daily  records  of  street  outreach  activities,  and 
tabulate  on  a  weekly  or  monthly  basis.  When  noticeable  increases  or 
decreases  occur  in  the  number  of  contacts  made,  record  the  reasons  for  these 
changes,  if  known.  Also,  leave  some  space  at  the  end  of  street  contact  forms 
for  outreach  workers  to  record  comments  made  by  contacts.  After  rapport  has 
been  established  with  some  street  contacts,  these  individuals  may  willingly 
offer  valuable  information  about  the  steps  they  have  taken  to  reduce  high-risk 
behaviors  such  as  seeking  counseling,  visiting  an  STD  clinic,  obtaining 
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condoms  and/or  bleach  kits,  or  about  their  efforts  to  share  AIDS  information 
with  their  sex  or  drug-using  partners. 

•  Role  plays.  Regular  assessment  of  staff,  particularly  those  working  directly 
with  target  groups,  is  also  important  for  ensuring  a  successful  program.  Make 
sure  they  understand  the  information  and  messages  to  be  communicated  to 
clients  on  the  streets;  ask  if  they  have  encountered  any  problems  or  have 
questions  concerning  their  work.  Periodically,  outreach  workers  can  perform 
role  plays  of  street  outreach  sessions  where  one  person  plays  a  client  and 
another  plays  the  outreach  worker.  Outside  observers  and  experts  should  be 
present  to  provide  feedback  on  delivery  style,  content,  and  accuracy  of 
messages. 

•  Informal  surveys.  It  is  usually  impossible  for  staff  to  follow  up  on  every 
street  outreach  contact.  Contacts  are  often  anonymous,  and  many  persons 
reached  may  have  no  fixed  address,  even  if  staff  had  the  time  to  look  for  them 
at  home.  When  it  is  possible  to  locate  individuals  who  have  previously 
received  information  and  who  feel  comfortable  with  staff,  have  staff  ask  them 
whether  the  outreach  effort  was  useful,  whether  they  learned  new  information, 
and  whether  they  took  some  recommended  action  —  sought  counseling  or 
have  begun  using  bleach  and  water  to  disinfect  used  needles. 

•  Key-person  interviews.  Ask  staff  of  local  drug  treatment  centers  and 
counseling  sites  to  find  out  how  new  clients  found  out  about  the  site  — 
whether  they  heard  about  it  through  street  outreach,  and  if  so,  what  are  the 
general  reactions  to  and  opinions  about  the  information  being  delivered  on  the 
street. 

C.  HIV  Counseling  and  Testing  Programs 

CDC  has  supported  the  widespread  implementation  of  HIV  counseling  and  testing 
programs  as  a  crucial  prevention  activity  to  reduce  the  spread  of  HIV  disease.  HIV 
counseling  and  testing  programs  are  based  on  the  assumption  that  most  individuals  will 
change  high-risk  behaviors  if  they  are  aware  of  their  serostatus  and  receive  appropriate 
counseling.9  Counseling  and  testing  services  usually  include  confidential  or  anonymous  HIV 
testing,  pre-  and  post-test  counseling,  and,  if  an  individual  tests  positive  for  HIV,  partner 
notification  and  referral. 

HIV  counseling  and  testing  serve  two  main  functions:  (1)  they  are  a  source  of 
primary  prevention  by  educating  uninfected  persons  about  assessing  and  reducing  their  risks, 
and  (2)  they  serve  as  a  source  of  secondary  prevention  by  counseling  infected  persons  about 
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maintaining  their  health  and  preventing  transmission  and  by  providing  referral  to  other 
services. 

Publicly-funded  counseling  and  testing  programs  offer  two  options  for  HIV  testing: 
confidential  or  anonymous  testing.  A  person  requesting  confidential  testing  consents  to 
having  his/her  test  result  become  part  of  his/her  medical  record.  There  are  several 
arguments  for  confidential  testing.  First,  confidential  testing  enables  testing  sites  to  contact 
and  inform  individuals  who  test  positive  for  HIV  but  do  not  return  for  test  results.  Second, 
doctors  theoretically  are  better  able  to  prescribe  treatments  for  HIV  and  other  illnesses  if  they 
are  fully  informed  about  a  patient’s  health.  Third,  partners  of  HIV-infected  individuals  can 
be  notified,  can  ask  questions,  and  can  obtain  information  and  referrals  to  other  services. 
Additionally,  HIV-infected  persons  can  take  advantage  of  early  intervention  services  by 
beginning  drug  therapy  or  becoming  involved  in  clinical  trial  programs.  When  persons 
choose  anonymous  testing,  individuals  are  assigned  numbers  only.  Names  are  not  obtained, 
and  test  results  cannot  be  included  in  the  medical  record.  Anonymous  testing  makes  it 
impossible  for  a  testing  site  to  contact  and  inform  a  person  who  tests  positive  for  HIV  and 
does  not  return  for  test  results.  However,  it  also  provides  absolute  protection  of 
confidentiality,  which  may  cause  more  people  to  obtain  testing. 

Key  Areas  for  Evaluation: 

When  evaluating  HIV  counseling  and  testing  programs,  assess  the  areas  which 
contribute  most  to  an  effective  service  delivery  system.  These  may  include  the  following: 

•  Adequacy  of  the  counseling  and  testing  procedures  —  reflecting  the  extent 
to  which  the  procedures  meet  client  needs,  which  include  confidentiality, 
reasonable  waiting  periods,  and  secure  linkages  between  counseling  and 
testing.  That  is,  HIV  counseling  must  always  precede  testing,  and  in  the  case 
of  positive  test  results,  must  follow  HIV  testing.  The  pre-test  counseling 
session  must  clearly  describe  the  testing  process  and  procedures  for  obtaining 
test  results.  The  time  delay  between  testing  and  availability  of  test  results  and 
post-test  counseling  should  be  as  short  as  possible  to  ensure  the  retention  of 
individuals  throughout  the  entire  testing  procedure;  the  provision  of  test  results 
should  be  confidential  and  free  of  barriers;  and  state/local  policies  on  partner 
notification  should  be  carefully  reviewed  and  followed. 

•  Adequacy  of  the  counseling  actually  provided  —  clients  often  need  support, 
reassurance,  sensitivity,  and  understanding  from  counselors;  a  risk  assessment 
is  also  important,  where  the  counselor  tries  to  elicit  key  information  about  the 
client’s  risk  behaviors  to  obtain  an  understanding  of  why  and  how  the  client  is 
at  risk  of  HIV/STDs  and  make  appropriate  referrals;  clients  should  also 
receive  accurate  and  appropriate  information  about  transmission  of  the  virus, 
risk  reduction  behaviors,  and  meaning  of  test  results.  For  clients  who  test 
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HIV-positive,  needs  such  as  coping  skills  —  how  to  accept  and  live  with 
HIV/AIDS  —  information  and  counseling  about  medical  care,  mental  health 
services,  partner  notification,  treatment  options,  and  clinical  trials  information 
are  especially  important. 

•  Proportion  of  clients  that  complete  all  counseling  and  testing  procedures  — 

this  includes  returning  for  test  results,  and  for  individuals  testing  HIV-positive, 
obtaining  post-test  counseling. 

•  Accessibility  of  services  —  determined  by  analyzing  barriers  to  services  such 
as  remote  location,  inconvenient  hours  of  service,  lack  of  bilingual  health  care 
providers  to  serve  limited-English-proficient  clients,  and  the  likelihood  of 
being  seen  by  friends  or  family. 

Evaluating  Project  Design: 

A  variety  of  evaluation  techniques  can  be  used  to  evaluate  current  HIV  counseling  and 
testing  methods,  systems,  and  written  materials.  These  include  the  following: 

•  Key-person  interviews.  Contact  other  already-established  testing  sites  and 
interview  their  key  staff  to  obtain  advice  and  suggestions  on  designing  and 
implementing  an  HIV  counseling  and  testing  program,  or  ask  for  feedback  on 
your  ideas  and  plans  for  designing  the  program;  find  out  what  obstacles  were 
incurred  in  the  early  planning  and  design  stages  and  how  they  were  resolved; 
also  ask  for  recommendations  on  other  places  and  people  to  contact. 

•  Expert  reviews.  Have  experts  from  STD  clinics,  community  health  centers, 
and  other  testing  sites  review  your  proposed  plan  and  make  suggestions  for 
improvement. 

•  Demonstration  sessions.  Schedule  a  day  or  two  for  conducting  demonstration 
sessions  to  assess  what  is  and  is  not  working  well  in  your  program.  During 
demonstration  sessions,  also  ask  clients  directly  or  through  surveys  about  their 
opinions  and  what  improvements  they  recommend. 

Evaluating  Project  Implementation  and  Outcomes: 

•  Documentation.  This  includes  ongoing  collection,  analysis,  and 
documentation  of  the  monthly  client  caseload,  number  of  HIV  tests  provided 
on  a  weekly  and/or  monthly  basis,  number  of  new  clients  per  month,  the 
percent  of  client  returns  for  test  results,  and  percent  of  clients  tested  who 
received  post-test  counseling,  number  of  partner  contacts,  and  referral 


NCLR  AIDS  Center 
Understanding  Evaluation  Techniques 


66 


information.  A  detailed  review  of  these  data  can  reveal  problems  which  need 
to  be  explored  further. 

Informal  Surveys.  Administer  short  one-page  informal  surveys  after  clients 
have  received  pre-test  counseling,  while  they  are  waiting  to  be  tested,  to  obtain 
their  thoughts,  opinions,  and  feedback  concerning  the  counseling  session  (e.g., 
whether  the  session  revealed  new  information,  whether  it  was  clear  and 
comprehensive,  whether  there  were  aspects  they  didn’t  like,  and  whether  the 
session  led  them  to  want  to  change  high-risk  behaviors).  Staff  should  collect 
surveys  from  clients  either  before  or  after  they  receive  their  tests.  Surveys 
could  also  be  handed  out  when  clients  come  in  to  pick  up  their  test  results. 
(This  would,  of  course,  only  be  appropriate  for  clients  who  test  HIV-negative.) 
The  information  collected  from  surveys  can  suggest  why  clients  may  not 
complete  all  counseling  and  testing  procedures  (e.g.,  test  results  take  too  long) 
and  suggest  ways  to  improve  the  process. 

Non-participant  observers.  Having  non-participant  observers  during  pre-test 
counseling  sessions  can  be  a  useful  evaluation  technique  since  pre-test 
counseling  sessions  are  generally  conducted  in  groups.  Observers  can  take 
notes  of  reactions  and  frequently-asked  questions  from  the  group,  and  consider 
some  of  the  following  questions  during  their  assessment: 

♦  What  is  the  message  being  given?  To  what  degree  is  testing,  risk 
reduction,  or  both  being  encouraged? 

♦  What  is  the  nature  and  quality  of  the  counselor/presenter  and  group 
interaction?  Does  the  counselor  demonstrate  good  presentation  skills? 

♦  How  are  counseling  messages  modified  for  different  clients/risk 
groups? 

♦  Are  counseling  messages  culturally  sensitive? 

It  is  very  useful  for  the  non-participant  observer  to  have  a  list  of  important 
points  the  counselor  is  expected  to  cover  during  the  session.  As  the  counselor 
covers  each  point  or  asks  important  questions,  the  observer  should  check  these 
items  off  the  list.  The  observer  should  also  record  important  questions  asked 
or  comments  made  by  the  clients  that  are  not  addressed  by  the  counselor 
during  a  standard  counseling  session.  At  the  end  of  the  session,  key  staff 
should  review  the  results  of  the  session.  Two  or  three  similar  sessions  should 
generally  be  observed  before  reaching  conclusions  and  providing  feedback  to 
the  counselor  and  his/her  supervisor. 
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•  Key-person  interviews.  Interview  heads  of  support  group  agencies  for  HIV¬ 
positive  persons,  gay  organizations,  drug  treatment  centers,  counseling  centers, 
etc.,  —  places  where  testing  sites  would  most  likely  refer  HIV-positive 
individuals  —  in  order  to  get  an  idea  of  the  number  of  persons  referred  to 
these  various  organizations  by  testing  sites.  This  will  provide  some  indication 
about  the  kind  of  information  being  communicated  to  HIV-positive  persons 
during  post-test  counseling  sessions. 

•  Planned  variation.  If  you  are  genuinely  uncertain  about  which  of  several 
counseling  approaches  is  more  appropriate,  implement  a  planned  variation  for 
a  two-month  period  by  testing  a  modified  procedure  against  the  standard 
procedure  to  find  out  which  works  better.  For  example,  the  modified 
procedure  might  be  to  provide  clients  of  testing  sites  new  information  during 
the  pre-test  counseling  session.  This  modified  approach  could  be  tested  for 
one  month  and  the  standard  approach  should  also  be  tested  for  one  month. 
Careful  documentation  of  the  number  of  participants  attending  pre-test 
counseling  sessions  and  number  of  clients  returning  for  test  results  the  week 
after  (HIV  test  results  are  usually  available  one  week  after  testing)  will  be 
especially  crucial  during  the  two  months  in  order  to  assure  an  accurate 
comparison  of  the  results  of  both  approaches.  At  the  end  of  the  two-month 
trial  period,  compare  the  number  of  client  returns  for  test  results  with  the 
number  of  clients  attending  pre-test  counseling  sessions  the  week  before  for 
each  trial  month.  The  approach  showing  the  most  client  returns  may  be  the 
most  favorable  approach  to  use  regularly.  However,  before  making  any  major 
changes  to  your  program,  test  the  approach  more  than  once. 
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VI.  CONCLUSION 


Hispanic  community-based  HIV/STD  programs  cannot  delay  the  implementation  and 
systematic  use  of  program  evaluation  techniques.  Community-based  organizations  must  take 
the  lead  in  compiling  information  and  disseminating  their  achievements  to  government 
officials,  funders,  and  other  local  community  groups.  This  will  provide  Congress,  policy 
makers,  and  the  general  public  a  greater  understanding  of  the  significance  of  and  need  for 
community-based  HIV/STD  programs. 

More  information  and  guidance  on  practical  methods  for  evaluating  HIV/STD 
education  and  risk  reduction  activities  are  badly  needed  in  the  Hispanic  community.  Much 
of  the  existing  information  is  inappropriate  for  the  kinds  of  HIV/STD-related  activities 
conducted  by  Hispanic  community-based  organizations  and  for  the  populations  they  serve. 
Evaluation  methods  are  often  complex,  even  overwhelming,  and  tend  to  concentrate  heavily 
on  costly  and  highly  structured  quantitative  research  rather  than  on  field-based,  lower-cost, 
qualitative  program  evaluation.  Needed  are  basic  "hands-on"  or  "cookbook-style"  evaluation 
manuals  that  can  help  remove  much  of  the  mystery  associated  with  evaluation.  This  guide  is 
intended  to  help  meet  this  need. 

A  major  priority  for  the  NCLR  AIDS  Center  is  to  collaborate  closely  with  Hispanic 
community-based  organizations  to  strengthen  their  capacity  to  evaluate  their  HIV/STD 
programs.  This  is  achieved  through  the  continuing  development  of  culturally  appropriate 
guides  and  manuals,  on-site  technical  assistance,  and  formal  evaluation  training  for 
community  groups.  This  is  the  second  in  a  series  of  evaluation  manuals  designed  for  use  by 
Hispanic  community-based  organizations.  A  third  manual  on  evaluation  is  planned. 

Hispanic  community-based  organizations  can  offer  valuable  assistance  in  the 
development  of  more  appropriate  information  on  evaluation.  By  testing  the  program 
evaluation  techniques  described  in  this  manual,  community-based  organizations  can  help  to 
identify  strengths  and  weaknesses  of  these  techniques  as  presented  and  suggest  modifications 
and  improvements.  Organizations  may  recommend  alternative  techniques  for  evaluating 
health  education  and  risk  reduction  activities  not  discussed  in  this  manual,  which  can  also 
benefit  other  community-based  programs.  This  information  can  only  be  obtained  by  testing 
various  evaluation  techniques,  documenting,  and  sharing  or  disseminating  their  results. 

Finally,  we  must  not  lose  sight  of  the  basic  mission  behind  all  HIV/STD-related 
activities  —  to  save  as  many  lives  as  possible  and  to  prevent  the  spread  of  HIV/STDs.  In  the 
absence  of  an  effective  cure  for  HIV/AIDS,  it  is  critical  to  identify  primary  prevention 
programs  or  approaches  that  show  benefits  in  motivating  individuals  to  abstain  from,  or 
reduce  levels  of,  high-risk  behaviors,  and  that  encourage  individuals  to  adopt  the  consistent 
use  of  practices  which  reduce  the  likelihood  of  HIV/STD  infection. 
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This  manual  describes  many  techniques  that  can  be  used  to  evaluate  the  effectiveness 
of  these  programs.  By  simply  taking  the  time  to  consider  which  techniques  are  most  suitable 
for  a  given  intervention,  and  then  coordinating  the  key  pieces  (e.g.,  staff,  target  group, 
materials,  etc.)  all  community-based  organizations  should  be  able  to  effectively  develop  and 
implement  an  evaluation  plan  for  their  HIV/STD  prevention  programs. 
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APPENDIX 

WORKSHEETS 


Worksheet  A:  Goals  and  Objectives.  The  purpose  of  this  form  is  to  record  key  project 
information  such  as  project  goals,  outcome  and  process  objectives,  and  project  deadlines. 
Other  information  to  record  on  this  form  is  a  description  of  the  project,  its  start  and  end 
date,  and  the  name  of  the  project  director.  This  form  should  be  updated  to  reflect  any 
revisions  made  to  the  project  (e.g.,  change  of  project  director,  revised  outcome  or  process 
objectives,  changes  in  deadlines,  etc.). 

Worksheet  B:  Community  Interview  Guide.  This  guide  is  useful  to  document  important 
information  about  a  community  interview  session.  Section  I  of  the  form  asks  for  a 
description  of  the  session  —  the  date,  time,  site,  number  and  description  of  participants,  and 
key  staff  members  involved  in  the  session.  Section  II  lists  sample  questions  which  the 
facilitator  can  ask  the  group. 

Worksheet  C:  HIV/STD  Education  Workshop  Post-Session  Assessment  Form.  Post¬ 
session  assessment  forms  provide  trainers  and  facilitators  of  workshops  and  training  seminars 
written  records  of  the  participants’  overall  assessment  of  the  training  session.  This  form  is 
handed  out  to  participants  once  the  workshop  or  training  seminar  has  ended. 

Worksheet  D:  Client  Encounter  Form.  This  form  documents  each  time  a  client  receives 
HIV/AIDS-related  services  from  an  agency.  It  records  such  information  as  the  type  of 
services  provided,  date  and  site  of  staff-client  encounter,  staff  persons  providing  services, 
and  results  of  the  encounter.  This  form  can  be  completed  by  staff  immediately  following  the 
client  encounter. 

Worksheet  E:  HIV/STD  Presentation  and  Workshop  Logging  Form.  The  purpose  of  this 
form  is  to  track  the  types  of  presentations  or  workshops  conducted  by  staff  of  HIV/STD 
programs.  The  form  asks  for  such  information  as  site  of  training,  characteristics  of 
participants,  and  level  of  group  involvement.  The  sections  on  General  and  Site  Information 
can  be  completed  by  staff  before  the  date  of  the  presentation  or  workshop  (whenever  this 
information  is  available).  The  remaining  information  can  be  completed  immediately 
following  the  session. 

Worksheet  F:  Street  Outreach  Contact  Sheet.  This  is  an  adaptation  of  a  form  ("street 
card")  used  by  Guadalupe  Center,  Inc.  of  Kansas  City,  Missouri.  The  street  card,  completed 
by  outreach  workers,  documents  key  information  about  each  street  contact,  including  time 
and  location  of  street  contact,  demographic  information,  type(s)  of  risk  behaviors  for 
HIV/STDs,  HIV  counseling  and  testing  information,  and  type  of  referral(s)  made  by  the 
outreach  worker. 
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Agency  Name 


WORKSHEET  A: 

GOALS  AND  OBJECTIVES 


Project  Name: 

Description  (briefly  describe  the  target  groups  and 
intervention  methods  of  the  project ): 

Project  Start  Date: 

Project  End  Date: 

Project  Director: 

I.  Goals 

List  the  major  goal(s)  ol  the  project. 

Goal  1: _ 


Goal  2: 
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II.  Objectives 

a.  List  the  outcome  objective(s)  of  the  project  AND  expected  date(s)  of 
completion. 

(This  states  what  will  be  different  as  a  result  of  the  project.) 

Example:  To  increase  the  knowledge  of  at  least  50  clients  attending  Elwood  STD  clinic  about  the  major 
transmission  modes  and  met  hods  for  reducing  the  riskforHIV  infection  within  six  months  after  project  initiation. 

1. 


2. 


3. 


b.  List  all  major  process  objectives  (or  activities)  AND  dates  of  completion  for 
each. 

Example:  To  develop  an  EIIV/STD  education  and  prevention  training  curriculum ,  including  agenda, exercises, 
and  reference  materials  within  one  month  after  project  initiation. 

Example:  To  conduct  three  HIV/STD  education  and  prevention  workshops  targeting  a  total  of  45  clients  from 
the  Elwood  STD  Cliiuc  within  three  months  after  project  initiation. 
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WORKSHEET  B: 

COMMUNITY  INTERVIEW  GUIDE 

I.  General  Information 

a.  Provide  a  brief  description  of  the  activity. 


b.  Date  of  activity: _ _ 

c.  Time: _  _ _ _ _  _ 

d.  Activity  Site: _ 

Address: _ 

e.  Provide  a  description  of  the  participants.  If  you  are  targeting  specific  groups, 
please  specify  the  group(s)  and  the  number  of  individuals  in  each  group  (e.g., 
number  of  teenagers,  number  of  Hispanic  women,  number  of  recovering 
addicts,  etc.): 


f.  Total  number  of  participants  (excluding  project  staff): _ 

g.  Briefly  describe  the  methods  used  for  recruiting  the  participants: 


h.  Language(s)  used  in  the  session: 
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i.  Provide  the  names  of  all  project  staff  involved  in  this  activity: 
Community  interview  coordinator(s): 


Facilitator!  s)/moderator(s): 


Recorder(s)/observer(s): 


II.  Interview  Questions 

( T  he  following  are  sample  questions  based  on  the  case  study  shown  on  page  31 .  The  facilitator  can  ask  the  group  these 
questions  once  the  session  has  ended.  The  observer  should  write  down  as  many  of  the  responses  to  the  questions  as 
possible.) 

1 .  NX/hat  is  this  skit  all  about? 

2.  NX/hat  kind  of  words  would  you  use  to  describe  the  skit? 

3.  Which  character  roles  did  you  like  best/least? 

4.  What  were  the  most  powerful  [emotionally  stirring]  scenes  during  the  skit? 

5.  Did  the  skit  portray  real-life  situations?  Why  or  why  not? 

6.  Did  the  skit  portray  real-life  characters?  Why  or  why  not? 

7.  What  message(s)  did  you  learn  from  this  skit? 

8.  How  would  you  apply  what  you  have  learned  from  this  skit  to  your  daily  life 
activities? 

9.  [Ask  students]  Would  you  recommend  showing  this  skit  to  other  high  school 
students?  Why  or  why  not? 

1  0.  Is  this  skit  useful  for  other  groups  besides  teenagers,  such  as  parents  and 
teachers?  Why  or  why  not? 

1 1 .  What  additional  character  roles  should  have  been  portrayed  in  the  skit? 

1  2.  What  changes  in  the  skit  would  you  recommend,  to  make  it  more  effective  in 
getting  its  message  across? 
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WORKSHEET  C: 

HIV/STD  EDUCATION  WORKSHOP 
POST-SESSION  ASSESSMENT  FORM 


In  order  lo  help  us  improve  future  training  workshops,  we  would  appreciate  your  feedback  on  this  training  workshop.  Please 
complete  this  brief  assessment  form  and  return  to  a  trainer  before  leaving  the  session.  Thank  you  for  your  assistance. 

1 .  How  much  did  you  know  about  HIV/STDs  before  this  training  workshop? 

□  A  lot  □  A  fair  amount 

□  A  little  □  None  at  all 

2.  How  would  you  rate  the  workshop  in  terms  of  its  overall  value  to  you  on  a 
scale  of  one  through  ten,  where  1  =  Poor  and  1  0  =  Excellent.  Please  circle 
one  number. 

1  2  3  4  5 

6  7  8  9  10 

3.  How  would  you  rate  the  following  aspects  of  the  workshop: 


Excellent 

Very 

(iood 

(iood 

Fair 

Poor 

a.  Training  site 

and  facilities? 

□ 

□ 

□ 

□ 

□ 

b.  Hand-outs 
and  take-home 
materials? 

□ 

□ 

□ 

□ 

□ 

c.  Organization  of 
materials  and 
hand-outs? 

□ 

□ 

□ 

□ 

□ 

d.  Trainer 

presentations? 

□ 

□ 

□ 

□ 

□ 

e.  Exercises 
and  participant 
interaction? 

□ 

□ 

□ 

□ 

□ 

f.  Content  of  the 

training? 

□ 

□ 

□ 

□ 

□ 

MIR 

NATIONAL  COUNCIL  OF  IA  RAZA 


NCLR  AIDS  Center 
HIV/STD 
Evaluation  Forms 


Agency  Name 


4.  After  completing  this  seminar,  can  you: 


Yes  No 

a.  Identify  the  major 

ways  HIV/STDs  is  passed 

on  to  another  person?  □  □ 

b.  Identify  “high-risk” 

behaviors  for  HIV/STDs?  □  □ 

c.  Explain  at  least  three 
ways  for  reducing  the 

risk  of  HIV/STDs?  □  □ 

d.  Discuss  the  major  differences 

between  HIV  and  AIDS?  G  G 

e.  Explain  the  “latency” 
period  between  HIV  infection 

and  full-blown  AIDS?  G  G 


f.  Advise  a  friend  about  the 
likelihood  of  getting 
HIV/AIDS  through  a  blood 

transfusion?  G  Q 

g.  Tell  friends  what  to  do 

if  they  think  they  have  been 

been  infected  with  HIV/STDs?  G  G 

h.  Explain  the  relationship 
between  non-HIV  sexually 
transmitted  diseases 

and  HIV?  G  G 


Unsure 


Q 

Q 

G 

G 

G 


G 

G 

Q 


5.  Will  you  change  any  behaviors  (e.g.,  sexual  and/or  drug  use)  that  may  cause 
you  to  become  infected  with  HIV/STDs  as  a  result  of  what  you  learned  during 
this  workshop? 

G  Yes  G  No  G  Unsure 


If  so,  how  will  you  change  behaviors? 


■ 
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6.  Will  you  try  to  educate  friends  and/or  family  about  the  ways  HIV/STDs  are 
passed  on  and  how  to  reduce  the  chances  of  becoming  infected? 

Q  Yes  □  No  Ql  Unsure 

7.  What  was  your  favorite  topic  or  section  during  this  training  workshop? 


8.  What  was  your  least  favorite  topic  or  section  during  this  workshop? 


9.  Would  you  recommend  this  workshop  to  your  friends  and/or  family? 

Q  Yes  □  No  Q  Unsure 


1  0.  Please  list  any  other  topics  that  should  be  included  in  this  workshop. 


1  1 .  Please  add  any  other  comments 
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WORKSHEET  D: 

CLIENT  ENCOUNTER  FORM 

Complete  this  form  for  each  one-on-one  client  encounter  session.  This  form  will  be  used  to  track  HTV/STD-reiated  services 
provided  to  clients.  When  completed,  place  form  in  client' s  file. 


1.  Date  of  Encounter: _  6.  Reason(s)  for  the  session: 

2.  Client  Name/I. D.  #: 


3.  Has  client  received  services 
from  this  agency  before? 

□  Yes  □  No 

If  yes,  what  was  the  date  of  the 
last  session? _ 

4.  Where  is  the  session  taking 
place?  Please  check  one: 

□  Counselor’s  office 

□  Client’s  Home 

□  Church 

□  School 

□  Other  (please  specify): _ 


5.  Who  initiated  the  session? 
Please  check  all  that  apply. 

□  Client 

□  Staff 

□  Family 

□  Spouse/Companion 

□  Other  (please  explain): _ 


7.  Were  others  present  at  the 
session  besides  the  client  and 
staff? 

□  Yes  □  No 

If  yes,  who  else  was  present 
(e.g.,  spouse,  family,  friend)? 


8.  What  type(s)  of  services  were 
provided  to  the  client  during 
this  session  (check  all  that 
apply)? 

□  HIV/AIDS  101  and  other  STD  education 

□  HIV/STD  testing 

□  Substance  abuse  counseling 

□  Information  and  referral  to  other  services 
within  the  agency  (e.g.,  support  groups, 
companion  programs,  etc.) 

List  types  of  services  referred  to: _ 
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□  Referrals  to  other  HIV/AIDS-related 
services  outside  of  the  agency  (e.g.,  HIV 
counseling  and  testing,  drug  and  alcohol 
treatment,  etc.) 

List  types  of  services  referred  to: _ 


□  Other  (please  explain):. 


9.  Has  client  followed  up  on 
referrals  made  at  the  last 
session? 

□  Yes  □  No 

If  yes,  what  action  did  client 
take? 


1  0.  Has  client  indicated  that  s/he 
has  made  any  behavior 
changes  to  reduce  the  risk  of 
HIV/STDs? 

□  Yes  □  No 

If  yes,  please  explain: _ 


11.1s  there  any  other  evidence  that 
client  is  more  knowledgeable 
about  HIV/STDs? 

Comments: 


1  2.  Has  a  follow-up  session  with 
client  been  scheduled? 

□  Yes  □  No 


1  3.  Total  Time  of  Session:  Hours _  Minutes 

1  4.  This  form  was  completed  by: _ 
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WORKSHEET  E: 

HIV/STD  PRESENTATION  AND  WORKSHOP 
LOGGING  FORM 

Staff  must  complete  this  form  for  every  HIV/STD  presentation  or  workshop  held. 

I.  General  Information 

a.  Name  of  PresentationAX/orkshop: _ 

b.  Date(s)  of  PresentationAX/orkshop: _ Time: 

c.  Length  of  Activity: _ 

d.  Briefly  describe  the  content  and  purpose  of  the  activity. 


e.  Trainers/Facilitators: 


f.  Language  of  PresentationAX/orkshop  (check  one): 

G  English  G  Spanish  G  Both 

II.  Site  Information 

a.  Name  of  Agency  or  Site  Location: _ 


b.  Address: 


c.  Telephone  #:  ( 


Contact  Person: 
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d.  Does  the  site  have  any  restrictions  (e.g.,  no  condom  distributions,  no  use  of 
specific  terms,  etc.)  which  presenters  must  adhere  to?  Please  explain. 


e.  Describe  the  types  of  written/visual  materials  used  during  presentation/ 
workshop. _ 


III.  Participant  Information 

a.  Describe  the  primary  participant  or  target  group  (e.g.,  all  males,  all  females, 
young  parents,  high  school  teenagers,  farmworkers,  etc.): _ 


b.  Number  of  Participants: 

_  Males  _ Females  _ Total 


Primary  Age  Group  (check  one): 

□  13-19  □  20-24 

□ 

25-29 

□  30-40  □  4 1  and  over 

□ 

Mixed  ages 

□ 

Unknown 

d.  Race/Ethnicity  (specify  the  number  of  individuals  in  each  racial/ethnic  group  if  known): 

_  Hispanic  — >  Indicate  the  number  within  each  Hispanic  subgroup,  it'  known: 

_  Black  (non-Hispanic)  _  Mexican  American  _  Central  American 

_  White  (non-Hispanic)  _  Puerto  Rican  _  South  American 

_  Other  (specify)  -  Cuban  -  Other 
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e.  Did  the  group  show  interest  in  the  workshop/presentation  (explain)? 


f.  Assess  the  level  of  group  participation  (e.g.,  did  everyone  participate,  only  a 
few? ) . _ 


g.  Assess  the  performance  of  participants  on  the  group  exercises  (e.g.,  were 
there  problems  understanding  the  exercises?  Were  they  able  to  complete  all 
exercises?). 


h.  How  many  participants  stayed  throughout  the  entire  workshop/presentation? 


What  specific  area(s)  did  the  group  have  problems  understanding? 


j.  What  specific  area(s)  or  topic(s)  did  the  group  seem  to  enjoy  most? 


k.  Were  pre-  and  post-test  surveys  administered? 

□  Yes  Q  No 


I.  Did  participants  have  problems  completing  these?  Explain. 
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WORKSHEET  F: 

STREET  OUTREACH  CONTACT  SHEET 

(Adapted  from  Street  Card  used  by  Guadalupe  Center.  Inc.) 

Outreach  Workers :  Complete  this  form  for  each  street  contact  made,  obtaining  as  much  of  the  information  as  possible.  Names 
and  addresses  of  contacts  are  optional. 

I.  General  Information 

a.  Worker  Initials: _ 

b.  Contact  #: _ 

c.  Date: _  _ 

d.  Time: _ 

e.  Location: _ 

II.  Contact  Description 

a.  Name  (optional) _ 

b.  Address  (optional) _ _  _ 


c.  Sex:  □  Male  □  Female 

d.  Approximate  Age _ 

e.  Race/Ethnicity _ 

III.  Substance  Use  (Check  all  that  apply) 


a. 

□  Injecting  drugs  □ 

Alcohol 

□ 

Other 

b. 

Frequency  of  Use:  □ 

Daily 

□ 

Weekly 

□ 

Monthly 
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IV.  Sexual  Behavior  (Check  all  that  apply) 


a. 

Multi-partners: 

□  Yes  □ 

No 

b. 

□ 

Heterosexual 

□  Homosexual 

□ 

Bisexual 

□  Other 

V.  Testing/Referral 

a.  Tested  for  HIV:  □  Yes  □  No 

b.  Referred  to: _ 

c.  Referred  for  (explain  type  of  assistance  needed): _ 


VI.  Comments: 


c 


J 
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